https://cstwiki.wtb.tue.nl/api.php?action=feedcontributions&user=S165296&feedformat=atomControl Systems Technology Group - User contributions [en]2024-03-28T23:17:44ZUser contributionsMediaWiki 1.39.5https://cstwiki.wtb.tue.nl/index.php?title=Mobile_Robot_Control_2021_Group_3&diff=116895Mobile Robot Control 2021 Group 32021-05-27T11:46:06Z<p>S165296: /* Path Planning */</p>
<hr />
<div><h1>Group Members</h1><br />
<br />
<h4>Students</h4><br />
<br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->[[File:Cecile.jpeg|180px]]<br />
|<!--col2-->[[File:LarsHagendoorn.jpg|125px]]<br />
|<!--col3-->[[File:WardKlip.jpg|125px]]<br />
|-<br />
|<!--col1-->Cecile Conrad<br />
|<!--col2-->Lars Hagendoorn<br />
|<!--col3-->Ward Klip<br />
|-<br />
|<!--col1-->1016008 <br />
|<!--col2-->1013526<br />
|<!--col3-->1015501 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |c.conrad@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |l.s.hagendoorn@student.tue.nl <br />
|<!--col3-->style="width: 150pt;" |w.m.klip@student.tue.nl<br />
|-<br />
|<!--col1-->[[File:Foto_Sjoerd.jpg|125px]]<br />
|<!--col2-->[[File:EduardSzucs.jpg|125px]]<br />
|<!--col3--><br />
|-<br />
|<!--col1-->Sjoerd Leemrijse<br />
|<!--col2-->Eduard-Istvan Szucs<br />
|<!--col3-->Matthijs Teurlings<br />
|-<br />
|<!--col1-->1009082 <br />
|<!--col2-->1536419 <br />
|<!--col3-->1504274 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |s.j.leemrijse@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |e.szucs@student.tue.nl<br />
|<!--col3-->style="width: 150pt;" |m.h.b.teurlings@student.tue.nl<br />
|}<!--end wikitable--><br />
<br />
<h4>Tutor</h4><br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->Jordy Senden<br />
|-<br />
|<!--col3-->style="width: 150pt;" |j.p.f.senden@tue.nl<br />
|}<!--end wikitable--><br />
<br><br />
<br />
= Introduction =<br />
Due to COVID-19 the pressure on the hospitals has increased enormously, exposing the ongoing shortage of medical personnel. This reduces the care which can be provided to the ones in need of medical attention. To reduce the workload of the nurses, robotic devices could be implemented to assist in e.g. the retrieval of the patients medicines. During this course PICO's software is developed with this purpose in mind. In the first part the basics of the software are displayed and tested during the escape room challenge. In the second part more detailed software designing is employed for the hospital challenge. <br />
<br />
<br />
= Design Document =<br />
To get an overview of the project, a design document was created, which can be found [[:File:MRC-Design-Document-Group-3.pdf|here]]. In this document, the requirements, the components and specifications, and the functions and interfaces are described. Following this, the content will be used to better understand what will be done in the Escape Room Challenge and the Hospital Challenge.<br />
<br />
<br />
= Escape Room Challenge =<br />
<br />
=== Introduction ===<br />
In this year's version of the course, we are going to use a simulation that reproduces the exact same behavior of the real PICO robot. <br />
<br />
The first major milestone of this course is the ''Escape Room Challenge'' in which we are faced with the task of driving our robot out of a square room, through a corridor. This environment is said to have walls that are not perfectly straight and corners that are not perfectly perpendicular. This requires a more robust design of our code, such that if, for example, the robot encounters a slightly different corner angle, it would still operate successfully. In order to achieve this goal, we can use the data that is coming from the laser scanner, as well as the data coming from the encoders attached to the wheels of the robot. <br />
<br />
We are given two trials to complete the challenge. A trial ends if our robot does at least one of the following actions:<br />
* Bumps into a wall, but a slight touch is allowed if the judges consider it acceptable;<br />
* Has not moved nor done any action for 30 seconds;<br />
* The total time spent in the room exceeds 5 minutes;<br />
<br />
The challenge is completed if the robot does not bump into any wall, respects the time limitations, and when the entire rear wheel of the robot has passed the finish line that is placed at least at 3 m into the corridor.<br />
<br />
=== The robot's behavior ===<br />
<br />
In the figure presented on the right, one can visualize the state machine after which the robot behavior is created for the ''Escape Room Challenge''. PICO starts by scanning its environment while turning a maximum of 180 degrees. Since PICO has a viewing range of 4 rad, corresponding to approximately 230 degrees, turning 180 degrees will result in a visualization of the entire environment. If during this rotation the corridor is detected, the middle of the corridor is set as a target. PICO will start driving toward the target, while continuously scanning and aligning with the middle of the corridor. Therefore, PICO will already be fully aligned with the entrance of the corridor upon arrival and it can simply drive into the corridor. If PICO gets too close to a wall in the corridor it will start following the wall algorithm. The wall algorithm will first align the robot with the closest wall and it will start driving alongside this wall. PICO only turns if it encounters a corner or another corridor. Also, it will continuously correct its position relative to the wall, meaning that if the detected wall is not entirely perpendicular to the robot's front direction, it will adjust its position accordingly. If PICO does not find a corridor during the initial scanning procedure, it will rotate away from the closest corner and start driving forward. While driving PICO keeps scanning and if it detects a corridor it will target the middle of the corridor and follow the same process described above. In the unfortunate case PICO reaches a wall, without detecting a corridor, it will start following the previously described wall algorithm. <br />
[[File:SM 1.png|600px|center|thumb| Figure 1: The state machine of the robot in the Escape Room Challenge.]]<br />
<br />
== Corridor detection ==<br />
[[File:LRF.PNG|500px|right|thumb|baseline| Figure 2: The laser scanner range (not the true scale).]]<br />
In the escape room challenge, the only goal is to detect a corridor and drive through it. With this in mind, obstacle detection and the orientation are not considered relevant for this specific challenge. The software should be able to:<br />
* Divide the LRF data into data segments. <br />
* Fit lines through the segments.<br />
* Use these lines to compute edges and corners.<br />
* Determine target positions.<br />
<br />
=== Data segmentation ===<br />
<h5>1. Wall Detection</h5><br />
First, distances outside the range of the LRF, corresponding to 0.1 - 10m, are removed. The remaining data point are transformed from polar to cartesian coordinates using: <br />
<br />
[[File:Formula - PolarToCart Coord 999.PNG|x40px]]<br />
<br />
where ''ρ<sub>i</sub>'' is the distance between PICO and an obstacle, ''θ<sub>i</sub>'' is the angle of the obstacle with regard to the robot, ''θ<sub>min</sub>''=-2 radians and is the lowest angle that can be measured, and ''i'' is the number of increments (0 to 999) where each increment is of size 4/1000.<br />
<br />
The data segmentation is done using the split and merge algorithm. This algorithm can be used to detect line segments, which in this case represent the wall. The first step is ''data segmentation'':<br />
# The indices of the first and last data point resulting from the LRF data are used as i_start and i_end. <br />
# Calculate the perpendicular distance from a line from i_start to i_end using: [[File:Formula Perp Dist.PNG|x50px]]<br />
# Find the maximum distance calculated in step 2 and its corresponding index. <br />
## If the distance is above a certain threshold: split the line. Now recursively go back to step 2 for both the segments before and after the index corresponding to the maximum distance. Keep repeating this process until all i_start and the used index in step 2 are adjacent. <br />
## If the distance is below a certain threshold: a line is found. Save the current i_start and i_end.<br />
<br />
After all the segments are found a line is drawn between the start and the end points of each individual segment. These lines can now be used for the computations of ''edges and corners''. <br />
<br />
<h5>2. Corridor and edge detection</h5><br />
As previously mentioned, in the Escape Room Challenge, one is faced with one room with one corridor. It is therefore key to classify the angles between two line segments as either convex or concave. In case the angle is concave PICO recognizes this as a corner and if it is convex it is classified as an edge. The two points which are the most convex are assumed to be the edges of the corridor. The target position is placed in the middle of these, i.e. it is assumed to be the entrance of the corridor. Worth noticing is that PICO keeps scanning while driving towards the previously set target, if it finds a more suitable corridor edge the target will be recalculated.<br />
<br />
== Results of the Escape Room Challenge == <br />
On May 12th 2021, all groups of the course 4SC020 Mobile Robot Control performed their fastest and/or most reliable version of how a PICO robot should exit a room through the only corridor present. Figure 3 shows the execution of the Escape Room Challenge by Group 3. As can be seen, the corridor through which the PICO needs to leave to finish is behind it. At the start, the robot turns counterclockwise to scan for the angles indicating a corridor. However, as can be seen more clearly in Figure 4, the robot thinks it has found a corridor right from the start. Though, this is actually not the case considering the corridor is outside of the LRF's field of view at that point in time. What PICO has found are the angles on the wall to the bottom left of the robot. It drives towards that, but since PICO is programmed to keep on scanning its surroundings, it is able to correct itself when it finds the sharper angles of the actual corridor. Following this, it aligns itself with the entrance of the corridor before driving towards the exit. Considering PICO is able to align itself with the middle of the entrance of the corridor, it can simply drive straight to the corridor. Moreover, because the corridor itself is not particularly narrow, PICO does not have to align itself with one of its walls.<br />
<br />
Following the prerequisites, during this execution, PICO did '''not''':<br />
* bump into any walls<br />
* stop moving after the start of the trial<br />
* spend more than 5 minutes inside the room<br />
<br />
What's more, Group 3 has completed this challenge in one try in approximately 13 seconds using the [https://gitlab.tue.nl/mobile-robot-control-group-2021/group3/-/tree/riskyAlgorithm Risky Algorithm]. With this time, Group 3 is the fastest of all groups and wins the Escape Room Challenge 2021 and has, thereby, successfully completed the first major milestone of the project. While this first algorithm was successful, a more safe algorithm was also provided as a back-up. <br />
<br />
<gallery widths=500px heights=300px><br />
File: 4SC020 EscRoom 480p.gif |Figure 3: The execution of the Escape Room Challenge by Group 3.<br />
File:4SC020 EscRoom Gr3 2021.jpg |Figure 4: Corridor 'found' at the start.<br />
</gallery><br />
<br />
= Hospital Challenge =<br />
==Path Planning==<br />
For the robot to reach every cabinet in a specified order it needs to know how to get from one cabinet to the next. Here path planning comes to the rescue. For the path planning different algorithms were investigated. One of the simplest path planning algorithms is Dijkstra's algorithms. This method starts from a initial vertex and calculates for all its surrounding neighbors the costs to get there. Sequentially are for all the neighbors the costs for their neighbors calculated and so on. This process is visualized in '''x''' . A drawback of this method is that a lot of costs are computed which are not needed, which makes the process computationally expensive. An improvement of the Dijkstra algorithm is the A* algorithm. In the A* algorithm a priority is given to nodes which have a lower distance to the goal than others. Both methods are represented in '''x'''. The A* method can become computationally expensive when there are e.g. high dimensions. This problem can be solved by using the rapidly-exploring random threes(RRT) algorithm. RRT randomly creates points and connects those to the nearest available node. Iteratively more nodes and thus more specific paths are generated. However, RRT by itself is most of the time not sufficient and an extra path planning algorithm is needed. Since for the hospital challenge the path planning should be quick and there is no indication for high dimension, the A* algorithm is used. <br />
<br />
<gallery widths=350px heights=300px><br />
File: Dijkstras_progress_animation.gif |Figure X: Dijkstra's algorithm<br />
File: Astar_progress.gif |Figure X: A* algorithm<br />
File: RRT graph1.png |Figure X:The RRT algorithm<br />
</gallery><br />
<br />
The A* algorithm applied will be grid based, meaning the global map will be transferred into a grid. In this grid a path needs to be planned between a starting node and a goal node. The algorithm starts from a starting node. I then computes for each surrounding node the G-cost, which is the distance from the starting node. Also the heuristic(H) cost is calculated, which is the distance to the end node. The total cost can then be simply calculated by adding the G and H cost. The algorithm then goes to the surrounding cell with the lowest total cost and for this node it evaluates all the surrounding nodes in a similar fashion. The node which is visited is then added to the closed list, to ensure the algorithm only visit each node once. All the evaluated surrounding nodes which are not yet visited are added to the open list, so the node with the lowest cost can be found easily if needed. Grid points with obstacles are placed in the closed list, so they will not be evaluated by the algorithm. I there are multiple nodes with the same total cost the one with the lowest H-cost (so closest to the goal) is picked. If there are multiple, one will be randomly picked. When the total costs of the surrounding cells start increasing the path is no longer the cheapest. The list with open nodes will then be scanned for the node with the lowest total cost and from this node the process will start again. In case the robot is enclosed with objects and thus no path can be found, the program will automatically exit the path planning while indicating that no path could be found. The visualization of our implemented A* algorithm can be seen in figure '''x'''<br />
<br />
<gallery widths=350px heights=300px><br />
File: program.gif |Figure X: A* algorithm<br />
</gallery><br />
<br />
<br />
<h2>Task Division</h2><br />
In order to show on what task each student from the group is, or was, working on, we made this Excel table in which every student entered their participation on a certain task. Please note that this table is often updated.<br />
<br />
If there is a need for updating or just visualizing the current state of the table, please access the following link: [[https://tuenl-my.sharepoint.com/:x:/r/personal/e_szucs_student_tue_nl/Documents/Task_division_group3.xlsx?d=w7ac7c99aa4f348e989fda57515ec0225&csf=1&web=1&e=ecSmeb| Task division.]] <br />
<br />
Everyone in this group has editing right and everyone else has just viewing right to the excel under the link found above.</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=Mobile_Robot_Control_2021_Group_3&diff=116894Mobile Robot Control 2021 Group 32021-05-27T11:42:30Z<p>S165296: /* Corridor detection */</p>
<hr />
<div><h1>Group Members</h1><br />
<br />
<h4>Students</h4><br />
<br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->[[File:Cecile.jpeg|180px]]<br />
|<!--col2-->[[File:LarsHagendoorn.jpg|125px]]<br />
|<!--col3-->[[File:WardKlip.jpg|125px]]<br />
|-<br />
|<!--col1-->Cecile Conrad<br />
|<!--col2-->Lars Hagendoorn<br />
|<!--col3-->Ward Klip<br />
|-<br />
|<!--col1-->1016008 <br />
|<!--col2-->1013526<br />
|<!--col3-->1015501 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |c.conrad@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |l.s.hagendoorn@student.tue.nl <br />
|<!--col3-->style="width: 150pt;" |w.m.klip@student.tue.nl<br />
|-<br />
|<!--col1-->[[File:Foto_Sjoerd.jpg|125px]]<br />
|<!--col2-->[[File:EduardSzucs.jpg|125px]]<br />
|<!--col3--><br />
|-<br />
|<!--col1-->Sjoerd Leemrijse<br />
|<!--col2-->Eduard-Istvan Szucs<br />
|<!--col3-->Matthijs Teurlings<br />
|-<br />
|<!--col1-->1009082 <br />
|<!--col2-->1536419 <br />
|<!--col3-->1504274 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |s.j.leemrijse@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |e.szucs@student.tue.nl<br />
|<!--col3-->style="width: 150pt;" |m.h.b.teurlings@student.tue.nl<br />
|}<!--end wikitable--><br />
<br />
<h4>Tutor</h4><br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->Jordy Senden<br />
|-<br />
|<!--col3-->style="width: 150pt;" |j.p.f.senden@tue.nl<br />
|}<!--end wikitable--><br />
<br><br />
<br />
= Introduction =<br />
Due to COVID-19 the pressure on the hospitals has increased enormously, exposing the ongoing shortage of medical personnel. This reduces the care which can be provided to the ones in need of medical attention. To reduce the workload of the nurses, robotic devices could be implemented to assist in e.g. the retrieval of the patients medicines. During this course PICO's software is developed with this purpose in mind. In the first part the basics of the software are displayed and tested during the escape room challenge. In the second part more detailed software designing is employed for the hospital challenge. <br />
<br />
<br />
= Design Document =<br />
To get an overview of the project, a design document was created, which can be found [[:File:MRC-Design-Document-Group-3.pdf|here]]. In this document, the requirements, the components and specifications, and the functions and interfaces are described. Following this, the content will be used to better understand what will be done in the Escape Room Challenge and the Hospital Challenge.<br />
<br />
<br />
= Escape Room Challenge =<br />
<br />
=== Introduction ===<br />
In this year's version of the course, we are going to use a simulation that reproduces the exact same behavior of the real PICO robot. <br />
<br />
The first major milestone of this course is the ''Escape Room Challenge'' in which we are faced with the task of driving our robot out of a square room, through a corridor. This environment is said to have walls that are not perfectly straight and corners that are not perfectly perpendicular. This requires a more robust design of our code, such that if, for example, the robot encounters a slightly different corner angle, it would still operate successfully. In order to achieve this goal, we can use the data that is coming from the laser scanner, as well as the data coming from the encoders attached to the wheels of the robot. <br />
<br />
We are given two trials to complete the challenge. A trial ends if our robot does at least one of the following actions:<br />
* Bumps into a wall, but a slight touch is allowed if the judges consider it acceptable;<br />
* Has not moved nor done any action for 30 seconds;<br />
* The total time spent in the room exceeds 5 minutes;<br />
<br />
The challenge is completed if the robot does not bump into any wall, respects the time limitations, and when the entire rear wheel of the robot has passed the finish line that is placed at least at 3 m into the corridor.<br />
<br />
=== The robot's behavior ===<br />
<br />
In the figure presented on the right, one can visualize the state machine after which the robot behavior is created for the ''Escape Room Challenge''. PICO starts by scanning its environment while turning a maximum of 180 degrees. Since PICO has a viewing range of 4 rad, corresponding to approximately 230 degrees, turning 180 degrees will result in a visualization of the entire environment. If during this rotation the corridor is detected, the middle of the corridor is set as a target. PICO will start driving toward the target, while continuously scanning and aligning with the middle of the corridor. Therefore, PICO will already be fully aligned with the entrance of the corridor upon arrival and it can simply drive into the corridor. If PICO gets too close to a wall in the corridor it will start following the wall algorithm. The wall algorithm will first align the robot with the closest wall and it will start driving alongside this wall. PICO only turns if it encounters a corner or another corridor. Also, it will continuously correct its position relative to the wall, meaning that if the detected wall is not entirely perpendicular to the robot's front direction, it will adjust its position accordingly. If PICO does not find a corridor during the initial scanning procedure, it will rotate away from the closest corner and start driving forward. While driving PICO keeps scanning and if it detects a corridor it will target the middle of the corridor and follow the same process described above. In the unfortunate case PICO reaches a wall, without detecting a corridor, it will start following the previously described wall algorithm. <br />
[[File:SM 1.png|600px|center|thumb| Figure 1: The state machine of the robot in the Escape Room Challenge.]]<br />
<br />
== Corridor detection ==<br />
[[File:LRF.PNG|500px|right|thumb|baseline| Figure 2: The laser scanner range (not the true scale).]]<br />
In the escape room challenge, the only goal is to detect a corridor and drive through it. With this in mind, obstacle detection and the orientation are not considered relevant for this specific challenge. The software should be able to:<br />
* Divide the LRF data into data segments. <br />
* Fit lines through the segments.<br />
* Use these lines to compute edges and corners.<br />
* Determine target positions.<br />
<br />
=== Data segmentation ===<br />
<h5>1. Wall Detection</h5><br />
First, distances outside the range of the LRF, corresponding to 0.1 - 10m, are removed. The remaining data point are transformed from polar to cartesian coordinates using: <br />
<br />
[[File:Formula - PolarToCart Coord 999.PNG|x40px]]<br />
<br />
where ''ρ<sub>i</sub>'' is the distance between PICO and an obstacle, ''θ<sub>i</sub>'' is the angle of the obstacle with regard to the robot, ''θ<sub>min</sub>''=-2 radians and is the lowest angle that can be measured, and ''i'' is the number of increments (0 to 999) where each increment is of size 4/1000.<br />
<br />
The data segmentation is done using the split and merge algorithm. This algorithm can be used to detect line segments, which in this case represent the wall. The first step is ''data segmentation'':<br />
# The indices of the first and last data point resulting from the LRF data are used as i_start and i_end. <br />
# Calculate the perpendicular distance from a line from i_start to i_end using: [[File:Formula Perp Dist.PNG|x50px]]<br />
# Find the maximum distance calculated in step 2 and its corresponding index. <br />
## If the distance is above a certain threshold: split the line. Now recursively go back to step 2 for both the segments before and after the index corresponding to the maximum distance. Keep repeating this process until all i_start and the used index in step 2 are adjacent. <br />
## If the distance is below a certain threshold: a line is found. Save the current i_start and i_end.<br />
<br />
After all the segments are found a line is drawn between the start and the end points of each individual segment. These lines can now be used for the computations of ''edges and corners''. <br />
<br />
<h5>2. Corridor and edge detection</h5><br />
As previously mentioned, in the Escape Room Challenge, one is faced with one room with one corridor. It is therefore key to classify the angles between two line segments as either convex or concave. In case the angle is concave PICO recognizes this as a corner and if it is convex it is classified as an edge. The two points which are the most convex are assumed to be the edges of the corridor. The target position is placed in the middle of these, i.e. it is assumed to be the entrance of the corridor. Worth noticing is that PICO keeps scanning while driving towards the previously set target, if it finds a more suitable corridor edge the target will be recalculated.<br />
<br />
== Results of the Escape Room Challenge == <br />
On May 12th 2021, all groups of the course 4SC020 Mobile Robot Control performed their fastest and/or most reliable version of how a PICO robot should exit a room through the only corridor present. Figure 3 shows the execution of the Escape Room Challenge by Group 3. As can be seen, the corridor through which the PICO needs to leave to finish is behind it. At the start, the robot turns counterclockwise to scan for the angles indicating a corridor. However, as can be seen more clearly in Figure 4, the robot thinks it has found a corridor right from the start. Though, this is actually not the case considering the corridor is outside of the LRF's field of view at that point in time. What PICO has found are the angles on the wall to the bottom left of the robot. It drives towards that, but since PICO is programmed to keep on scanning its surroundings, it is able to correct itself when it finds the sharper angles of the actual corridor. Following this, it aligns itself with the entrance of the corridor before driving towards the exit. Considering PICO is able to align itself with the middle of the entrance of the corridor, it can simply drive straight to the corridor. Moreover, because the corridor itself is not particularly narrow, PICO does not have to align itself with one of its walls.<br />
<br />
Following the prerequisites, during this execution, PICO did '''not''':<br />
* bump into any walls<br />
* stop moving after the start of the trial<br />
* spend more than 5 minutes inside the room<br />
<br />
What's more, Group 3 has completed this challenge in one try in approximately 13 seconds using the [https://gitlab.tue.nl/mobile-robot-control-group-2021/group3/-/tree/riskyAlgorithm Risky Algorithm]. With this time, Group 3 is the fastest of all groups and wins the Escape Room Challenge 2021 and has, thereby, successfully completed the first major milestone of the project. While this first algorithm was successful, a more safe algorithm was also provided as a back-up. <br />
<br />
<gallery widths=500px heights=300px><br />
File: 4SC020 EscRoom 480p.gif |Figure 3: The execution of the Escape Room Challenge by Group 3.<br />
File:4SC020 EscRoom Gr3 2021.jpg |Figure 4: Corridor 'found' at the start.<br />
</gallery><br />
<br />
= Hospital Challenge =<br />
==Path Planning==<br />
For the robot to reach every cabinet in a specified order it needs to know how to get from one cabinet to the next. Here path planning comes to the rescue. For the path planning different algorithms were investigated. One of the simplest path planning algorithms is Dijkstra's algorithms. This method starts from a initial vertex and calculates for all its surrounding neighbors the costs to get there. Sequentially are for all the neighbors the costs for their neighbors calculated and so on. This process is visualized in '''x''' . A drawback of this method is that a lot of costs are computed which are not needed, which makes the process computationally expensive. An improvement of the Dijkstra algorithm is the A* algorithm. In the A* algorithm a priority is given to nodes which have a lower distance to the goal than others. Both methods are represented in '''x'''. The A* method can become computationally expensive when there are e.g. high dimensions. This problem can be solved by using the rapidly-exploring random threes(RRT) algorithm. RRT randomly creates points and connects those to the nearest available node. Iteratively more nodes and thus more specific paths are generated. However, RRT by itself is most of the time not sufficient and an extra path planning algorithm is needed. Since for the hospital challenge the map the path planning should be quick and there is no indication for high dimension the A* algorithm is used. <br />
<br />
<gallery widths=350px heights=300px><br />
File: Dijkstras_progress_animation.gif |Figure X: Dijkstra's algorithm<br />
File: Astar_progress.gif |Figure X: A* algorithm<br />
File: RRT graph1.png |Figure X:The RRT algorithm<br />
</gallery><br />
<br />
The A* algorithm applied will be grid based, meaning the global map will be transferred into a grid. In this grid a path needs to be planned between a starting node and a goal node. The algorithm starts from a starting node. I then computes for each surrounding node the G-cost, which is the distance from the starting node. Also the heuristic(H) cost is calculated, which is the distance to the end node. The total cost can then be simply calculated by adding the G and H cost. The algorithm then goes to the surrounding cell with the lowest total cost and for this node it evaluates all the surrounding nodes in a similar fashion. The node which is visited is then added to the closed list, to ensure the algorithm only visit each node once. All the evaluated surrounding nodes which are not yet visited are added to the open list, so the node with the lowest cost can be found easily if needed. Grid points with obstacles are placed in the closed list, so they will not be evaluated by the algorithm. I there are multiple nodes with the same total cost the one with the lowest H-cost (so closest to the goal) is picked. If there are multiple, one will be randomly picked. When the total costs of the surrounding cells start increasing the path is no longer the cheapest. The list with open nodes will then be scanned for the node with the lowest total cost and from this node the process will start again. In case the robot is enclosed with objects and thus no path can be found, the program will automatically exit the path planning while indicating that no path could be found. The visualization of our implemented A* algorithm can be seen in figure '''x'''<br />
<br />
<gallery widths=350px heights=300px><br />
File: program.gif |Figure X: A* algorithm<br />
</gallery><br />
<br />
<br />
<h2>Task Division</h2><br />
In order to show on what task each student from the group is, or was, working on, we made this Excel table in which every student entered their participation on a certain task. Please note that this table is often updated.<br />
<br />
If there is a need for updating or just visualizing the current state of the table, please access the following link: [[https://tuenl-my.sharepoint.com/:x:/r/personal/e_szucs_student_tue_nl/Documents/Task_division_group3.xlsx?d=w7ac7c99aa4f348e989fda57515ec0225&csf=1&web=1&e=ecSmeb| Task division.]] <br />
<br />
Everyone in this group has editing right and everyone else has just viewing right to the excel under the link found above.</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=Mobile_Robot_Control_2021_Group_3&diff=116893Mobile Robot Control 2021 Group 32021-05-27T11:42:13Z<p>S165296: /* Corridor detection */</p>
<hr />
<div><h1>Group Members</h1><br />
<br />
<h4>Students</h4><br />
<br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->[[File:Cecile.jpeg|180px]]<br />
|<!--col2-->[[File:LarsHagendoorn.jpg|125px]]<br />
|<!--col3-->[[File:WardKlip.jpg|125px]]<br />
|-<br />
|<!--col1-->Cecile Conrad<br />
|<!--col2-->Lars Hagendoorn<br />
|<!--col3-->Ward Klip<br />
|-<br />
|<!--col1-->1016008 <br />
|<!--col2-->1013526<br />
|<!--col3-->1015501 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |c.conrad@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |l.s.hagendoorn@student.tue.nl <br />
|<!--col3-->style="width: 150pt;" |w.m.klip@student.tue.nl<br />
|-<br />
|<!--col1-->[[File:Foto_Sjoerd.jpg|125px]]<br />
|<!--col2-->[[File:EduardSzucs.jpg|125px]]<br />
|<!--col3--><br />
|-<br />
|<!--col1-->Sjoerd Leemrijse<br />
|<!--col2-->Eduard-Istvan Szucs<br />
|<!--col3-->Matthijs Teurlings<br />
|-<br />
|<!--col1-->1009082 <br />
|<!--col2-->1536419 <br />
|<!--col3-->1504274 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |s.j.leemrijse@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |e.szucs@student.tue.nl<br />
|<!--col3-->style="width: 150pt;" |m.h.b.teurlings@student.tue.nl<br />
|}<!--end wikitable--><br />
<br />
<h4>Tutor</h4><br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->Jordy Senden<br />
|-<br />
|<!--col3-->style="width: 150pt;" |j.p.f.senden@tue.nl<br />
|}<!--end wikitable--><br />
<br><br />
<br />
= Introduction =<br />
Due to COVID-19 the pressure on the hospitals has increased enormously, exposing the ongoing shortage of medical personnel. This reduces the care which can be provided to the ones in need of medical attention. To reduce the workload of the nurses, robotic devices could be implemented to assist in e.g. the retrieval of the patients medicines. During this course PICO's software is developed with this purpose in mind. In the first part the basics of the software are displayed and tested during the escape room challenge. In the second part more detailed software designing is employed for the hospital challenge. <br />
<br />
<br />
= Design Document =<br />
To get an overview of the project, a design document was created, which can be found [[:File:MRC-Design-Document-Group-3.pdf|here]]. In this document, the requirements, the components and specifications, and the functions and interfaces are described. Following this, the content will be used to better understand what will be done in the Escape Room Challenge and the Hospital Challenge.<br />
<br />
<br />
= Escape Room Challenge =<br />
<br />
=== Introduction ===<br />
In this year's version of the course, we are going to use a simulation that reproduces the exact same behavior of the real PICO robot. <br />
<br />
The first major milestone of this course is the ''Escape Room Challenge'' in which we are faced with the task of driving our robot out of a square room, through a corridor. This environment is said to have walls that are not perfectly straight and corners that are not perfectly perpendicular. This requires a more robust design of our code, such that if, for example, the robot encounters a slightly different corner angle, it would still operate successfully. In order to achieve this goal, we can use the data that is coming from the laser scanner, as well as the data coming from the encoders attached to the wheels of the robot. <br />
<br />
We are given two trials to complete the challenge. A trial ends if our robot does at least one of the following actions:<br />
* Bumps into a wall, but a slight touch is allowed if the judges consider it acceptable;<br />
* Has not moved nor done any action for 30 seconds;<br />
* The total time spent in the room exceeds 5 minutes;<br />
<br />
The challenge is completed if the robot does not bump into any wall, respects the time limitations, and when the entire rear wheel of the robot has passed the finish line that is placed at least at 3 m into the corridor.<br />
<br />
=== The robot's behavior ===<br />
<br />
In the figure presented on the right, one can visualize the state machine after which the robot behavior is created for the ''Escape Room Challenge''. PICO starts by scanning its environment while turning a maximum of 180 degrees. Since PICO has a viewing range of 4 rad, corresponding to approximately 230 degrees, turning 180 degrees will result in a visualization of the entire environment. If during this rotation the corridor is detected, the middle of the corridor is set as a target. PICO will start driving toward the target, while continuously scanning and aligning with the middle of the corridor. Therefore, PICO will already be fully aligned with the entrance of the corridor upon arrival and it can simply drive into the corridor. If PICO gets too close to a wall in the corridor it will start following the wall algorithm. The wall algorithm will first align the robot with the closest wall and it will start driving alongside this wall. PICO only turns if it encounters a corner or another corridor. Also, it will continuously correct its position relative to the wall, meaning that if the detected wall is not entirely perpendicular to the robot's front direction, it will adjust its position accordingly. If PICO does not find a corridor during the initial scanning procedure, it will rotate away from the closest corner and start driving forward. While driving PICO keeps scanning and if it detects a corridor it will target the middle of the corridor and follow the same process described above. In the unfortunate case PICO reaches a wall, without detecting a corridor, it will start following the previously described wall algorithm. <br />
[[File:SM 1.png|600px|center|thumb| Figure 1: The state machine of the robot in the Escape Room Challenge.]]<br />
<br />
== Corridor detection ==<br />
[[File:LRF.PNG|500px|right|thumb|baseline| Figure 2: The laser scanner range (not the true scale).]]<br />
In the escape room challenge, the only goal is to detect a corridor and drive through it. With this in mind, obstacle detection and the orientation are not considered relevant for this specific challenge. The software should be able to:<br />
* Divide the LRF data into data segments. <br />
* Fit lines through the segments.<br />
* Use these lines to compute edges and corners.<br />
* Determine target positions.<br />
<br />
=== Data segmentation ===<br />
<h5>1. Wall Detection</h5><br />
First, distances outside the range of the LRF, corresponding to 0.1 - 10m, are removed. The remaining data point are transformed from polar to cartesian coordinates using: <br />
<br />
[[File:Formula - PolarToCart Coord 999.PNG|x40px]]<br />
<br />
where ''ρ<sub>i</sub>'' is the distance between PICO and an obstacle, ''θ<sub>i</sub>'' is the angle of the obstacle with regard to the robot, ''θ<sub>min</sub>''=-2 radians and is the lowest angle that can be measured, and ''i'' is the number of increments (0 to 999) where each increment is of size 4/1000.<br />
<br />
The data segmentation is done using the split and merge algorithm. This algorithm can be used to detect line segments, which in this case represent the wall. The first step is ''data segmentation'':<br />
# The indices of the first and last data point resulting from the LRF data are used as i_start and i_end. <br />
# Calculate the perpendicular distance from a line from i_start to i_end using: [[File:Formula Perp Dist.PNG|x50px]]<br />
# Find the maximum distance calculated in step 2 and its corresponding index. <br />
## If the distance is above a certain threshold: split the line. Now recursively go back to step 2 for both the segments before and after the index corresponding to the maximum distance. Keep repeating this process until all i_start and the used index in step 2 are adjacent. <br />
## If the distance is below a certain threshold: a line is found. Save the current i_start and i_end.<br />
<br />
After all the segments are found a line is drawn between the start and the end points of each individual segment. These lines can now be used for the computations of ''edges and corners''. <br />
<br />
<h5>2. Corridor and edge detection</h5><br />
As previously mentioned, in the Escape Room Challenge, one is faced with one room with one corridor. It is therefore key to classify the angles between two line segments as either convex or concave. In case the angle is concave PICO recognizes this as a corner and if it is convex it is classified as an edge. The two points which are the most convex are assumed to be thee edges of the corridor. The target position is placed in the middle of these, i.e. it is assumed to be the entrance of the corridor. Worth noticing is that PICO keeps scanning while driving towards the previously set target, if it finds a more suitable corridor edge the target will be recalculated.<br />
<br />
== Results of the Escape Room Challenge == <br />
On May 12th 2021, all groups of the course 4SC020 Mobile Robot Control performed their fastest and/or most reliable version of how a PICO robot should exit a room through the only corridor present. Figure 3 shows the execution of the Escape Room Challenge by Group 3. As can be seen, the corridor through which the PICO needs to leave to finish is behind it. At the start, the robot turns counterclockwise to scan for the angles indicating a corridor. However, as can be seen more clearly in Figure 4, the robot thinks it has found a corridor right from the start. Though, this is actually not the case considering the corridor is outside of the LRF's field of view at that point in time. What PICO has found are the angles on the wall to the bottom left of the robot. It drives towards that, but since PICO is programmed to keep on scanning its surroundings, it is able to correct itself when it finds the sharper angles of the actual corridor. Following this, it aligns itself with the entrance of the corridor before driving towards the exit. Considering PICO is able to align itself with the middle of the entrance of the corridor, it can simply drive straight to the corridor. Moreover, because the corridor itself is not particularly narrow, PICO does not have to align itself with one of its walls.<br />
<br />
Following the prerequisites, during this execution, PICO did '''not''':<br />
* bump into any walls<br />
* stop moving after the start of the trial<br />
* spend more than 5 minutes inside the room<br />
<br />
What's more, Group 3 has completed this challenge in one try in approximately 13 seconds using the [https://gitlab.tue.nl/mobile-robot-control-group-2021/group3/-/tree/riskyAlgorithm Risky Algorithm]. With this time, Group 3 is the fastest of all groups and wins the Escape Room Challenge 2021 and has, thereby, successfully completed the first major milestone of the project. While this first algorithm was successful, a more safe algorithm was also provided as a back-up. <br />
<br />
<gallery widths=500px heights=300px><br />
File: 4SC020 EscRoom 480p.gif |Figure 3: The execution of the Escape Room Challenge by Group 3.<br />
File:4SC020 EscRoom Gr3 2021.jpg |Figure 4: Corridor 'found' at the start.<br />
</gallery><br />
<br />
= Hospital Challenge =<br />
==Path Planning==<br />
For the robot to reach every cabinet in a specified order it needs to know how to get from one cabinet to the next. Here path planning comes to the rescue. For the path planning different algorithms were investigated. One of the simplest path planning algorithms is Dijkstra's algorithms. This method starts from a initial vertex and calculates for all its surrounding neighbors the costs to get there. Sequentially are for all the neighbors the costs for their neighbors calculated and so on. This process is visualized in '''x''' . A drawback of this method is that a lot of costs are computed which are not needed, which makes the process computationally expensive. An improvement of the Dijkstra algorithm is the A* algorithm. In the A* algorithm a priority is given to nodes which have a lower distance to the goal than others. Both methods are represented in '''x'''. The A* method can become computationally expensive when there are e.g. high dimensions. This problem can be solved by using the rapidly-exploring random threes(RRT) algorithm. RRT randomly creates points and connects those to the nearest available node. Iteratively more nodes and thus more specific paths are generated. However, RRT by itself is most of the time not sufficient and an extra path planning algorithm is needed. Since for the hospital challenge the map the path planning should be quick and there is no indication for high dimension the A* algorithm is used. <br />
<br />
<gallery widths=350px heights=300px><br />
File: Dijkstras_progress_animation.gif |Figure X: Dijkstra's algorithm<br />
File: Astar_progress.gif |Figure X: A* algorithm<br />
File: RRT graph1.png |Figure X:The RRT algorithm<br />
</gallery><br />
<br />
The A* algorithm applied will be grid based, meaning the global map will be transferred into a grid. In this grid a path needs to be planned between a starting node and a goal node. The algorithm starts from a starting node. I then computes for each surrounding node the G-cost, which is the distance from the starting node. Also the heuristic(H) cost is calculated, which is the distance to the end node. The total cost can then be simply calculated by adding the G and H cost. The algorithm then goes to the surrounding cell with the lowest total cost and for this node it evaluates all the surrounding nodes in a similar fashion. The node which is visited is then added to the closed list, to ensure the algorithm only visit each node once. All the evaluated surrounding nodes which are not yet visited are added to the open list, so the node with the lowest cost can be found easily if needed. Grid points with obstacles are placed in the closed list, so they will not be evaluated by the algorithm. I there are multiple nodes with the same total cost the one with the lowest H-cost (so closest to the goal) is picked. If there are multiple, one will be randomly picked. When the total costs of the surrounding cells start increasing the path is no longer the cheapest. The list with open nodes will then be scanned for the node with the lowest total cost and from this node the process will start again. In case the robot is enclosed with objects and thus no path can be found, the program will automatically exit the path planning while indicating that no path could be found. The visualization of our implemented A* algorithm can be seen in figure '''x'''<br />
<br />
<gallery widths=350px heights=300px><br />
File: program.gif |Figure X: A* algorithm<br />
</gallery><br />
<br />
<br />
<h2>Task Division</h2><br />
In order to show on what task each student from the group is, or was, working on, we made this Excel table in which every student entered their participation on a certain task. Please note that this table is often updated.<br />
<br />
If there is a need for updating or just visualizing the current state of the table, please access the following link: [[https://tuenl-my.sharepoint.com/:x:/r/personal/e_szucs_student_tue_nl/Documents/Task_division_group3.xlsx?d=w7ac7c99aa4f348e989fda57515ec0225&csf=1&web=1&e=ecSmeb| Task division.]] <br />
<br />
Everyone in this group has editing right and everyone else has just viewing right to the excel under the link found above.</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=Mobile_Robot_Control_2021_Group_3&diff=115688Mobile Robot Control 2021 Group 32021-05-16T08:53:55Z<p>S165296: /* Hospital Challenge */</p>
<hr />
<div><h1>Group Members</h1><br />
<br />
<h4>Students</h4><br />
<br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->[[File:Cecile.jpeg|180px]]<br />
|<!--col2-->[[File:LarsHagendoorn.jpg|125px]]<br />
|<!--col3-->[[File:WardKlip.jpg|125px]]<br />
|-<br />
|<!--col1-->Cecile Conrad<br />
|<!--col2-->Lars Hagendoorn<br />
|<!--col3-->Ward Klip<br />
|-<br />
|<!--col1-->1016008 <br />
|<!--col2-->1013526<br />
|<!--col3-->1015501 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |c.conrad@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |l.s.hagendoorn@student.tue.nl <br />
|<!--col3-->style="width: 150pt;" |w.m.klip@student.tue.nl<br />
|-<br />
|<!--col1-->[[File:Foto_Sjoerd.jpg|125px]]<br />
|<!--col2-->[[File:EduardSzucs.jpg|125px]]<br />
|<!--col3--><br />
|-<br />
|<!--col1-->Sjoerd Leemrijse<br />
|<!--col2-->Eduard-Istvan Szucs<br />
|<!--col3-->Matthijs Teurlings<br />
|-<br />
|<!--col1-->1009082 <br />
|<!--col2-->1536419 <br />
|<!--col3-->1504274 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |s.j.leemrijse@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |e.szucs@student.tue.nl<br />
|<!--col3-->style="width: 150pt;" |m.h.b.teurlings@student.tue.nl<br />
|}<!--end wikitable--><br />
<br />
<h4>Tutor</h4><br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->Jordy Senden<br />
|-<br />
|<!--col3-->style="width: 150pt;" |j.p.f.senden@tue.nl<br />
|}<!--end wikitable--><br />
<br><br />
<br />
= Introduction =<br />
Due to COVID-19 the pressure on the hospitals has increased enormously, exposing the ongoing shortage of medical personnel. This reduces the care which can be provided to the ones in need of medical attention. To reduce the workload of the nurses, robotic devices could be implemented to assist in e.g. the retrieval of the patients medicines. During this course PICO's software is developed with this purpose in mind. In the first part the basics of the software are displayed and tested during the escape room challenge. In the second part more detailed software designing is employed for the hospital challenge. <br />
<br />
<br />
= Design Document =<br />
To get an overview of the project, a design document was created, which can be found [[:File:MRC-Design-Document-Group-3.pdf|here]]. In this document, the requirements, the components and specifications, and the functions and interfaces are described. Following this, the content will be used to better understand what will be done in the Escape Room Challenge and the Hospital Challenge.<br />
<br />
<br />
= Escape Room Challenge =<br />
<br />
=== Introduction ===<br />
In this year's version of the course, we are going to use a simulation that reproduces the exact same behavior of the real PICO robot. <br />
<br />
The first major milestone of this course is the ''Escape Room Challenge'' in which we are faced with the task of driving our robot out of a square room, through a corridor. This environment is said to have walls that are not perfectly straight and corners that are not perfectly perpendicular. This requires a more robust design of our code, such that if, for example, the robot encounters a slightly different corner angle, it would still operate successfully. In order to achieve this goal, we can use the data that is coming from the laser scanner, as well as the data coming from the encoders attached to the wheels of the robot. <br />
<br />
We are given two trials to complete the challenge. A trial ends if our robot does at least one of the following actions:<br />
* Bumps into a wall, but a slight touch is allowed if the judges consider it acceptable;<br />
* Has not moved nor done any action for 30 seconds;<br />
* The total time spent in the room exceeds 5 minutes;<br />
<br />
The challenge is completed if the robot does not bump into any wall, respects the time limitations, and when the entire rear wheel of the robot has passed the finish line that is placed at least at 3 m into the corridor.<br />
<br />
=== The robot's behavior ===<br />
<br />
In the figure presented on the right, one can visualize the state machine after which the robot behavior is created for the ''Escape Room Challenge''. PICO starts by scanning its environment while turning a maximum of 180 degrees. Since PICO has a viewing range of 4 rad, corresponding to approximately 230 degrees, turning 180 degrees will result in a visualization of the entire environment. If during this rotation the corridor is detected, the middle of the corridor is set as a target. PICO will start driving toward the target, while continuously scanning and aligning with the middle of the corridor. Therefore, PICO will already be fully aligned with the entrance of the corridor upon arrival and it can simply drive into the corridor. If PICO gets too close to a wall in the corridor it will start following the wall algorithm. The wall algorithm will first align the robot with the closest wall and it will start driving alongside this wall. PICO only turns if it encounters a corner or another corridor. Also, it will continuously correct its position relative to the wall, meaning that if the detected wall is not entirely perpendicular to the robot's front direction, it will adjust its position accordingly. If PICO does not find a corridor during the initial scanning procedure, it will rotate away from the closest corner and start driving forward. While driving PICO keeps scanning and if it detects a corridor it will target the middle of the corridor and follow the same process described above. In the unfortunate case PICO reaches a wall, without detecting a corridor, it will start following the previously described wall algorithm. <br />
[[File:SM 1.png|600px|center|thumb| Figure 1: The state machine of the robot in the Escape Room Challenge.]]<br />
<br />
== Corridor detection ==<br />
[[File:LRF.PNG|500px|right|thumb|baseline| Figure 2: The laser scanner range (not the true scale).]]<br />
In the escape room challenge, the only goal is to detect a corridor and drive through it. With this in mind, obstacle detection and the orientation are not considered relevant for this specific challenge. The software should be able to:<br />
* Divide the LRF data into data segments. <br />
* Fit lines through the segments.<br />
* Use these lines to compute edges and corners.<br />
* Determine target positions.<br />
<br />
=== Data segmentation ===<br />
<h5>1. Wall Detection</h5><br />
First, distances outside the range of the LRF, corresponding to 0.1 - 10m, are removed. The remaining data point are transformed from polar to cartesian coordinates using: <br />
<br />
[[File:Formula - PolarToCart Coord 999.PNG|x40px]]<br />
<br />
where ''ρ<sub>i</sub>'' is the distance between PICO and an obstacle, ''θ<sub>i</sub>'' is the angle of the obstacle with regard to the robot, ''θ<sub>min</sub>''=-2 radians and is the lowest angle that can be measured, and ''i'' is the number of increments (0 to 999) where each increment is of size 4/1000.<br />
<br />
The data segmentation is done using the split and merge algorithm. This algorithm can be used to detect line segments, which in this case represent the wall. The first step is ''data segmentation'':<br />
# The indices of the first and last data point resulting from the LRF data are used as i_start and i_end. <br />
# Calculate the perpendicular distance from a line from i_start to i_end using: [[File:Formula Perp Dist.PNG|x50px]]<br />
# Find the maximum distance calculated in step 2 and its corresponding index. <br />
## If the distance is above a certain threshold: split the line. Now recursively go back to step 2 for both the segments before and after the index corresponding to the maximum distance. Keep repeating this process until all i_start and the used index in step 2 are adjacent. <br />
## If the distance is below a certain threshold: a line is found. Save the current i_start and i_end.<br />
<br />
After all the segments are found a line is drawn between the start and the end points of each individual segment. These lines can now be used for the computations of ''edges and corners''. <br />
<br />
<h5>2. Corridor and edge detection</h5><br />
As previously mentioned, in the Escape Room Challenge, one is faced with one room with one corridor. It is therefore key to classify the angles between two line segments as either convex or concave. In case the angle is concave PICO recognizes this as a corner and if it is convex it is classified as an edge. The two points which are the most convex are assumed to be de edges of the corridor. The target position is placed in the middle of these, i.e. it is assumed to be the entrance of the corridor. Worth noticing is that PICO keeps scanning while driving towards the previously set target, if it finds a more suitable corridor edge the target will be recalculated.<br />
<br />
== Results of the Escape Room Challenge == <br />
On May 12th 2021, all groups of the course 4SC020 Mobile Robot Control performed their fastest and/or most reliable version of how a PICO robot should exit a room through the only corridor present. Figure 3 shows the execution of the Escape Room Challenge by Group 3. As can be seen, the corridor through which the PICO needs to leave to finish is behind it. At the start, the robot turns counterclockwise to scan for the angles indicating a corridor. However, as can be seen more clearly in Figure 4, the robot thinks it has found a corridor right from the start. Though, this is actually not the case considering the corridor is outside of the LRF's field of view at that point in time. What PICO has found are the angles on the wall to the bottom left of the robot. It drives towards that, but since PICO is programmed to keep on scanning its surroundings, it is able to correct itself when it finds the sharper angles of the actual corridor. Following this, it aligns itself with the entrance of the corridor before driving towards the exit. Considering PICO is able to align itself with the middle of the entrance of the corridor, it can simply drive straight to the corridor. Moreover, because the corridor itself is not particularly narrow, PICO does not have to align itself with one of its walls.<br />
<br />
Following the prerequisites, during this execution, PICO did '''not''':<br />
* bump into any walls<br />
* stop moving after the start of the trial<br />
* spend more than 5 minutes inside the room<br />
<br />
What's more, Group 3 has completed this challenge in one try in approximately 13 seconds using the [https://gitlab.tue.nl/mobile-robot-control-group-2021/group3/-/tree/riskyAlgorithm Risky Algorithm]. With this time, Group 3 is the fastest of all groups and wins the Escape Room Challenge 2021 and has, thereby, successfully completed the first major milestone of the project. While this first algorithm was successful, a more safe algorithm was also provided as a back-up. <br />
<br />
<gallery widths=500px heights=300px><br />
File: 4SC020 EscRoom 480p.gif |Figure 3: The execution of the Escape Room Challenge by Group 3.<br />
File:4SC020 EscRoom Gr3 2021.jpg |Figure 4: Corridor 'found' at the start.<br />
</gallery><br />
<br />
= Hospital Challenge =<br />
<br />
<h2>Task Division</h2><br />
In order to show on what task each student from the group is, or was, working on, we made this Excel table in which every student entered their participation on a certain task. Please note that this table is often updated.<br />
<br />
If there is a need for updating or just visualizing the current state of the table, please access the following link: [[https://tuenl-my.sharepoint.com/:x:/r/personal/e_szucs_student_tue_nl/Documents/Task_division_group3.xlsx?d=w7ac7c99aa4f348e989fda57515ec0225&csf=1&web=1&e=ecSmeb| Task division.]] <br />
<br />
Everyone in this group has editing right and everyone else has just viewing right to the excel under the link found above.</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=Mobile_Robot_Control_2021_Group_3&diff=115683Mobile Robot Control 2021 Group 32021-05-16T08:47:11Z<p>S165296: /* Introduction */</p>
<hr />
<div><h1>Group Members</h1><br />
<br />
<h4>Students</h4><br />
<br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->[[File:Cecile.jpeg|180px]]<br />
|<!--col2-->[[File:LarsHagendoorn.jpg|125px]]<br />
|<!--col3-->[[File:WardKlip.jpg|125px]]<br />
|-<br />
|<!--col1-->Cecile Conrad<br />
|<!--col2-->Lars Hagendoorn<br />
|<!--col3-->Ward Klip<br />
|-<br />
|<!--col1-->1016008 <br />
|<!--col2-->1013526<br />
|<!--col3-->1015501 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |c.conrad@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |l.s.hagendoorn@student.tue.nl <br />
|<!--col3-->style="width: 150pt;" |w.m.klip@student.tue.nl<br />
|-<br />
|<!--col1-->[[File:Foto_Sjoerd.jpg|125px]]<br />
|<!--col2-->[[File:EduardSzucs.jpg|125px]]<br />
|<!--col3--><br />
|-<br />
|<!--col1-->Sjoerd Leemrijse<br />
|<!--col2-->Eduard-Istvan Szucs<br />
|<!--col3-->Matthijs Teurlings<br />
|-<br />
|<!--col1-->1009082 <br />
|<!--col2-->1536419 <br />
|<!--col3-->1504274 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |s.j.leemrijse@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |e.szucs@student.tue.nl<br />
|<!--col3-->style="width: 150pt;" |m.h.b.teurlings@student.tue.nl<br />
|}<!--end wikitable--><br />
<br />
<h4>Tutor</h4><br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->Jordy Senden<br />
|-<br />
|<!--col3-->style="width: 150pt;" |j.p.f.senden@tue.nl<br />
|}<!--end wikitable--><br />
<br><br />
<br />
= Introduction =<br />
Due to COVID-19 the pressure on the hospitals has increased enormously, exposing the ongoing shortage of medical personnel. This reduces the care which can be provided to the ones in need of medical attention. To reduce the workload of the nurses, robotic devices could be implemented to assist in e.g. the retrieval of the patients medicines. During this course PICO's software is developed with this purpose in mind. In the first part the basics of the software are displayed and tested during the escape room challenge. In the second part more detailed software designing is employed for the hospital challenge. <br />
<br />
<br />
= Design Document =<br />
To get an overview of the project, a design document was created, which can be found [[:File:MRC-Design-Document-Group-3.pdf|here]]. In this document, the requirements, the components and specifications, and the functions and interfaces are described. Following this, the content will be used to better understand what will be done in the Escape Room Challenge and the Hospital Challenge.<br />
<br />
<br />
= Escape Room Challenge =<br />
<br />
=== Introduction ===<br />
In this year's version of the course, we are going to use a simulation that reproduces the exact same behavior of the real PICO robot. <br />
<br />
The first major milestone of this course is the ''Escape Room Challenge'' in which we are faced with the task of driving our robot out of a square room, through a corridor. This environment is said to have walls that are not perfectly straight and corners that are not perfectly perpendicular. This requires a more robust design of our code, such that if, for example, the robot encounters a slightly different corner angle, it would still operate successfully. In order to achieve this goal, we can use the data that is coming from the laser scanner, as well as the data coming from the encoders attached to the wheels of the robot. <br />
<br />
We are given two trials to complete the challenge. A trial ends if our robot does at least one of the following actions:<br />
* Bumps into a wall, but a slight touch is allowed if the judges consider it acceptable;<br />
* Has not moved nor done any action for 30 seconds;<br />
* The total time spent in the room exceeds 5 minutes;<br />
<br />
The challenge is completed if the robot does not bump into any wall, respects the time limitations, and when the entire rear wheel of the robot has passed the finish line that is placed at least at 3 m into the corridor.<br />
<br />
=== The robot's behavior ===<br />
<br />
In the figure presented on the right, one can visualize the state machine after which the robot behavior is created for the ''Escape Room Challenge''. PICO starts by scanning its environment while turning a maximum of 180 degrees. Since PICO has a viewing range of 4 rad, corresponding to approximately 230 degrees, turning 180 degrees will result in a visualization of the entire environment. If during this rotation the corridor is detected, the middle of the corridor is set as a target. PICO will start driving toward the target, while continuously scanning and aligning with the middle of the corridor. Therefore, PICO will already be fully aligned with the entrance of the corridor upon arrival and it can simply drive into the corridor. If PICO gets too close to a wall in the corridor it will start following the wall algorithm. The wall algorithm will first align the robot with the closest wall and it will start driving alongside this wall. PICO only turns if it encounters a corner or another corridor. Also, it will continuously correct its position relative to the wall, meaning that if the detected wall is not entirely perpendicular to the robot's front direction, it will adjust its position accordingly. If PICO does not find a corridor during the initial scanning procedure, it will rotate away from the closest corner and start driving forward. While driving PICO keeps scanning and if it detects a corridor it will target the middle of the corridor and follow the same process described above. In the unfortunate case PICO reaches a wall, without detecting a corridor, it will start following the previously described wall algorithm. <br />
[[File:SM 1.png|600px|center|thumb| Figure 1: The state machine of the robot in the Escape Room Challenge.]]<br />
<br />
== Corridor detection ==<br />
[[File:LRF.PNG|500px|right|thumb|baseline| Figure 2: The laser scanner range (not the true scale).]]<br />
In the escape room challenge, the only goal is to detect a corridor and drive through it. With this in mind, obstacle detection and the orientation are not considered relevant for this specific challenge. The software should be able to:<br />
* Divide the LRF data into data segments. <br />
* Fit lines through the segments.<br />
* Use these lines to compute edges and corners.<br />
* Determine target positions.<br />
<br />
=== Data segmentation ===<br />
<h5>1. Wall Detection</h5><br />
First, distances outside the range of the LRF, corresponding to 0.1 - 10m, are removed. The remaining data point are transformed from polar to cartesian coordinates using: <br />
<br />
[[File:Formula - PolarToCart Coord 999.PNG|x40px]]<br />
<br />
where ''ρ<sub>i</sub>'' is the distance between PICO and an obstacle, ''θ<sub>i</sub>'' is the angle of the obstacle with regard to the robot, ''θ<sub>min</sub>''=-2 radians and is the lowest angle that can be measured, and ''i'' is the number of increments (0 to 999) where each increment is of size 4/1000.<br />
<br />
The data segmentation is done using the split and merge algorithm. This algorithm can be used to detect line segments, which in this case represent the wall. The first step is ''data segmentation'':<br />
# The indices of the first and last data point resulting from the LRF data are used as i_start and i_end. <br />
# Calculate the perpendicular distance from a line from i_start to i_end using: [[File:Formula Perp Dist.PNG|x50px]]<br />
# Find the maximum distance calculated in step 2 and its corresponding index. <br />
## If the distance is above a certain threshold: split the line. Now recursively go back to step 2 for both the segments before and after the index corresponding to the maximum distance. Keep repeating this process until all i_start and the used index in step 2 are adjacent. <br />
## If the distance is below a certain threshold: a line is found. Save the current i_start and i_end.<br />
<br />
After all the segments are found a line is drawn between the start and the end points of each individual segment. These lines can now be used for the computations of ''edges and corners''. <br />
<br />
<h5>2. Corridor and edge detection</h5><br />
As previously mentioned, in the Escape Room Challenge, one is faced with one room with one corridor. It is therefore key to classify the angles between two line segments as either convex or concave. In case the angle is concave PICO recognizes this as a corner and if it is convex it is classified as an edge. The two points which are the most convex are assumed to be de edges of the corridor. The target position is placed in the middle of these, i.e. it is assumed to be the entrance of the corridor. Worth noticing is that PICO keeps scanning while driving towards the previously set target, if it finds a more suitable corridor edge the target will be recalculated.<br />
<br />
== Results of the Escape Room Challenge == <br />
On May 12th 2021, all groups of the course 4SC020 Mobile Robot Control performed their fastest and/or most reliable version of how a PICO robot should exit a room through the only corridor present. Figure 3 shows the execution of the Escape Room Challenge by Group 3. As can be seen, the corridor through which the PICO needs to leave to finish is behind it. At the start, the robot turns counterclockwise to scan for the angles indicating a corridor. However, as can be seen more clearly in Figure 4, the robot thinks it has found a corridor right from the start. Though, this is actually not the case considering the corridor is outside of the LRF's field of view at that point in time. What PICO has found are the angles on the wall to the bottom left of the robot. It drives towards that, but since PICO is programmed to keep on scanning its surroundings, it is able to correct itself when it finds the sharper angles of the actual corridor. Following this, it aligns itself with the entrance of the corridor before driving towards the exit. Considering PICO is able to align itself with the middle of the entrance of the corridor, it can simply drive straight to the corridor. Moreover, because the corridor itself is not particularly narrow, PICO does not have to align itself with one of its walls.<br />
<br />
Following the prerequisites, during this execution, PICO did '''not''':<br />
* bump into any walls<br />
* stop moving after the start of the trial<br />
* spend more than 5 minutes inside the room<br />
<br />
What's more, Group 3 has completed this challenge in one try in approximately 13 seconds using the [https://gitlab.tue.nl/mobile-robot-control-group-2021/group3/-/tree/riskyAlgorithm Risky Algorithm]. With this time, Group 3 is the fastest of all groups and wins the Escape Room Challenge 2021 and has, thereby, successfully completed the first major milestone of the project. While this first algorithm was successful, a more safe algorithm was also provided as a back-up. <br />
<br />
<gallery widths=500px heights=300px><br />
File: 4SC020 EscRoom 480p.gif |Figure 3: The execution of the Escape Room Challenge by Group 3.<br />
File:4SC020 EscRoom Gr3 2021.jpg |Figure 4: Corridor 'found' at the start.<br />
</gallery><br />
<br />
= Hospital Challenge =<br />
<br />
<h2>Task Division</h2><br />
In order to show on what task each student from the group is, or was, working on, we made this excel table in which every student entered their participation on a certain task. Please note that this table is often updated.<br />
<br />
If there is a need for updating or just visualizing the current state of the table, please access the following link: [[https://tuenl-my.sharepoint.com/:x:/r/personal/e_szucs_student_tue_nl/Documents/Task_division_group3.xlsx?d=w7ac7c99aa4f348e989fda57515ec0225&csf=1&web=1&e=ecSmeb| Task division.]] <br />
<br />
Everyone in this group has editing right and everyone else has just viewing right to the excel under the link found above.</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=Mobile_Robot_Control_2021_Group_3&diff=114969Mobile Robot Control 2021 Group 32021-05-04T13:14:22Z<p>S165296: </p>
<hr />
<div><h2>Group Members</h2><br />
<br />
<h4>Students</h4><br />
<br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->[[File:Cecile.jpeg|180px]]<br />
|<!--col2-->[[File:LarsHagendoorn.jpg|125px]]<br />
|<!--col3-->[[File:WardKlip.jpg|125px]]<br />
|-<br />
|<!--col1-->Cecile Conrad<br />
|<!--col2-->Lars Hagendoorn<br />
|<!--col3-->Ward Klip<br />
|-<br />
|<!--col1-->1016008 <br />
|<!--col2-->1013526<br />
|<!--col3-->1015501 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |c.conrad@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |l.s.hagendoorn@student.tue.nl <br />
|<!--col3-->style="width: 150pt;" |w.m.klip@student.tue.nl<br />
|-<br />
|<!--col1-->[[File:Foto_Sjoerd.jpg|125px]]<br />
|<!--col2-->[[File:EduardSzucs.jpg|125px]]<br />
|<!--col3--><br />
|-<br />
|<!--col1-->Sjoerd Leemrijse<br />
|<!--col2-->Eduard-Istvan Szucs<br />
|<!--col3-->Matthijs Teurlings<br />
|-<br />
|<!--col1-->1009082 <br />
|<!--col2-->1536419 <br />
|<!--col3-->1504274 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |s.j.leemrijse@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |e.szucs@student.tue.nl<br />
|<!--col3-->style="width: 150pt;" |m.h.b.teurlings@student.tue.nl<br />
|}<!--end wikitable--><br />
<br />
<h4>Tutor</h4><br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->Jordy Senden<br />
|-<br />
|<!--col3-->style="width: 150pt;" |j.p.f.senden@tue.nl<br />
|}<!--end wikitable--><br />
<br><br />
<br />
<h2>Design Document</h2><br />
<br />
<h2>Escape Room Challenge</h2><br />
<br />
<h2>Hospital Challenge</h2></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=Mobile_Robot_Control_2021_Group_3&diff=114968Mobile Robot Control 2021 Group 32021-05-04T13:14:00Z<p>S165296: </p>
<hr />
<div><h2>Group Members</h2><br />
<br />
<h4>Students</h4><br />
<br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->[[File:Cecile.jpeg|180px]]<br />
|<!--col2-->[[File:LarsHagendoorn.jpg|125px]]<br />
|<!--col3-->[[File:WardKlip.jpg|125px]]<br />
|-<br />
|<!--col1-->Cecile Conrad<br />
|<!--col2-->Lars Hagendoorn<br />
|<!--col3-->Ward Klip<br />
|-<br />
|<!--col1-->1016008 <br />
|<!--col2-->1013526<br />
|<!--col3-->1015501 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |c.conrad@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |l.s.hagendoorn@student.tue.nl <br />
|<!--col3-->style="width: 150pt;" |w.m.klip@student.tue.nl<br />
|-<br />
|<!--col1-->[[File:FotoSjoerd.jpg|125px]]<br />
|<!--col2-->[[File:EduardSzucs.jpg|125px]]<br />
|<!--col3--><br />
|-<br />
|<!--col1-->Sjoerd Leemrijse<br />
|<!--col2-->Eduard-Istvan Szucs<br />
|<!--col3-->Matthijs Teurlings<br />
|-<br />
|<!--col1-->1009082 <br />
|<!--col2-->1536419 <br />
|<!--col3-->1504274 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |s.j.leemrijse@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |e.szucs@student.tue.nl<br />
|<!--col3-->style="width: 150pt;" |m.h.b.teurlings@student.tue.nl<br />
|}<!--end wikitable--><br />
<br />
<h4>Tutor</h4><br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->Jordy Senden<br />
|-<br />
|<!--col3-->style="width: 150pt;" |j.p.f.senden@tue.nl<br />
|}<!--end wikitable--><br />
<br><br />
<br />
<h2>Design Document</h2><br />
<br />
<h2>Escape Room Challenge</h2><br />
<br />
<h2>Hospital Challenge</h2></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=Mobile_Robot_Control_2021_Group_3&diff=114967Mobile Robot Control 2021 Group 32021-05-04T13:13:21Z<p>S165296: </p>
<hr />
<div><h2>Group Members</h2><br />
<br />
<h4>Students</h4><br />
<br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->[[File:Cecile.jpeg|180px]]<br />
|<!--col2-->[[File:LarsHagendoorn.jpg|125px]]<br />
|<!--col3-->[[File:WardKlip.jpg|125px]]<br />
|-<br />
|<!--col1-->Cecile Conrad<br />
|<!--col2-->Lars Hagendoorn<br />
|<!--col3-->Ward Klip<br />
|-<br />
|<!--col1-->1016008 <br />
|<!--col2-->1013526<br />
|<!--col3-->1015501 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |c.conrad@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |l.s.hagendoorn@student.tue.nl <br />
|<!--col3-->style="width: 150pt;" |w.m.klip@student.tue.nl<br />
|-<br />
|<!--col1-->[[File:fotoSjoerd.jpg|125px]]<br />
|<!--col2-->[[File:EduardSzucs.jpg|125px]]<br />
|<!--col3--><br />
|-<br />
|<!--col1-->Sjoerd Leemrijse<br />
|<!--col2-->Eduard-Istvan Szucs<br />
|<!--col3-->Matthijs Teurlings<br />
|-<br />
|<!--col1-->1009082 <br />
|<!--col2-->1536419 <br />
|<!--col3-->1504274 <br />
|-<br />
|<!--col1-->style="width: 150pt;" |s.j.leemrijse@student.tue.nl<br />
|<!--col2-->style="width: 150pt;" |e.szucs@student.tue.nl<br />
|<!--col3-->style="width: 150pt;" |m.h.b.teurlings@student.tue.nl<br />
|}<!--end wikitable--><br />
<br />
<h4>Tutor</h4><br />
{| class="wikitable"<br />
|-<br />
|<!--col1-->Jordy Senden<br />
|-<br />
|<!--col3-->style="width: 150pt;" |j.p.f.senden@tue.nl<br />
|}<!--end wikitable--><br />
<br><br />
<br />
<h2>Design Document</h2><br />
<br />
<h2>Escape Room Challenge</h2><br />
<br />
<h2>Hospital Challenge</h2></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=File:Foto_Sjoerd.jpg&diff=114966File:Foto Sjoerd.jpg2021-05-04T13:12:16Z<p>S165296: </p>
<hr />
<div></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=Weekly_Log_PRE2019_3_Group15&diff=92765Weekly Log PRE2019 3 Group152020-04-06T19:28:45Z<p>S165296: /* Weekly Log */</p>
<hr />
<div>Go back to [[PRE2019 3 Group15]]<br />
<br />
=Weekly Log=<br />
==Week 2==<br />
<b>Goal</b>: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[PRE2019_3_Group15#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[PRE2019_3_Group15#References|(Gates, Subramanian & Gutwin, 2006)]], [[PRE2019_3_Group15#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summarized it. [[PRE2019_3_Group15#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[PRE2019_3_Group15#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[PRE2019_3_Group15#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[PRE2019_3_Group15#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[PRE2019_3_Group15#Milestones|milestones]] and [[PRE2019_3_Group15#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[PRE2019_3_Group15#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[PRE2019_3_Group15#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br />
==Week 3==<br />
<b>Goal</b>: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[PRE2019_3_Group15#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[PRE2019_3_Group15#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| Added to the SotA by implenting parts of the research <i> 30 minutes</i><br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br />
===Carnaval Break===<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[PRE2019_3_Group15#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br />
==Week 4==<br />
<b>Goal</b>: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[PRE2019_3_Group15#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[PRE2019_3_Group15#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br />
==Week 5==<br />
<b>Goal</b>: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[PRE2019_3_Group15#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>1 hour</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[PRE2019_3_Group15#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[PRE2019_3_Group15#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[PRE2019_3_Group15#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br />
==Week 6==<br />
<b>Goal</b>: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[PRE2019_3_Group15#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[PRE2019_3_Group15#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[PRE2019_3_Group15#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[PRE2019_3_Group15#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 3 hours </i><br />
|-<br />
|}<br />
<br />
==Week 7==<br />
<b>Goal</b>: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| Backend development + CSS slider development <i>7 hours</i><br />
| Did research on MATLAB Compiler for Java application: <i>3 hours</i><br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| Learned CSS + started integration of backend <i>6 hours</i><br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Updated the API reqiest code: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Tried to transform MATLAB functions to Java: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| Worked on a [[PRE2019_3_Group15#A GUI for QET matching|GUI]] for QET matching: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| Worked on the UI <i>6 hours</i><br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| Worked on [[PRE2019_3_Group15#A GUI for QET matching|GUI]] for QET matching: <i>3 hours</i><br />
|<br />
|<br />
| Made mockups of the lay-out of the final site and got feedback from users <i> 2.5 hours </i><br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| Reported my progression on the wiki and editing existing pages: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>1.5 hour</i><br/>Starting on validation of user needs: <i>1.5 hour</i><br />
| Worked on the UI <i>5 hours</i><br />
| <br />
|-<br />
|}<br />
<br />
==Week 8==<br />
<b>Goal</b>: Finish wiki, presentation<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 30-03<br />
| We attended the tutor session and had a skype meeting after that ourselves <br/><i>30 minutes + 1 hour</i><br />
| Attended the tutor meeting: <i>1.5 hours</i> <br />
| Attended the tutor meeting: <i>1.5 hours</i> <br />
| Attended the tutor meeting: <i>1.5 hours</i><br />
| Attended the tutor meeting: <i>1.5 hours</i><br />
| Attended the tutor meeting, slightly adjusted mockups and analyzed my own results from the user test. <i>2.5 hours</i> <br />
|-<br />
! scope="row"| Tuesday 31-03<br />
|<br />
| <br />
| Worked on the Wiki page: <i>1.5 hours</i><br />
| <br />
| Worked on restructuring the wiki <i> 2 hours </i><br />
| Updated the [[PRE2019_3_Group15##State of the Art | SotA]] and [[PRE2019_3_Group15##User interface | user interface]] reorganized the structure of the wiki <i> 3 hours </i> Worked on the new user interface of the website using CSS <i> 3 hours</i><br />
|-<br />
! scope="row"| Wednesday 01-04<br />
| <br />
|<br />
| Working on the presentation and wiki: <i>2 hours</i><br />
|<br />
| Read other wikis to find out what could still be added to our wiki. Executed user-tests <i> 1 hour </i><br />
| Worked on the user interface of the website using CSS <i> 1.5 hours </i><br />
|-<br />
! scope="row"| Thursday 02-04<br />
| Had an online group meeting: <i>1.5 hours</i><br />
|<br />
| Attended meeting: <i>1.5 hours</i>, worked on presentation: <i>2 hours</i><br />
| Attended the tutor meeting: <i>1.5 hours</i><br />
|Attended the group meeting: <i>1.5 hours</i><br />
|<br />
|-<br />
! scope="row"| Friday 03-04<br />
| <br />
| <br />
| Worked on the presentation: <i>1 hour</i>, had a meeting: <i>1 hour</i><br />
| <br />
| Worked on the [[PRE2019_3_Group15#Introduction | Introduction ]] and more on the structure of the page <i>1 hour</i><br />
| <br />
|-<br />
! scope="row"| Saturday 04-04<br />
| <br />
| <br />
| Worked on the presentation: <i>30 minutes</i><br />
|<br />
| Worked on the [[PRE2019_3_Group15#Validation of the user needs | Validation of the user needs ]] <i>2 hours</i><br />
| <br />
|-<br />
! scope="row"| Sunday 05-04<br />
| <br />
| <br />
| <br />
| Worked on the [[PRE2019_3_Group15#Validation of the user needs | Validation of the user needs ]] <i>4 hours</i><br />
| <br />
| Worked on the User Interface of the website. <i> 4 hours </i><br />
|-<br />
|}<br />
<br />
==Week 9==<br />
<b>Goal</b>: Finish wiki, site and last things<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 06-04<br />
| <br />
|<br />
| Worked on the presentation: <i>2 hours</i><br />
|<br />
|<br />
|Finished the UI of the website. Added a part on the UI in the wiki and did a spell check of the whole wiki <i> 7 hours + 1 hour</i><br />
|-</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=90135PRE2019 3 Group152020-03-31T10:06:47Z<p>S165296: /* System overview */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Introduction=<br />
==Problem Statement and Objectives==<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
==Approach, Milestones, and Deliverables==<br />
===Approach===<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
===<span style="background:yellow">Milestones</span>===<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
===Deliverables===<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
Danceability relates to the extent to which it is easy to dance on the music. This is a feature, described by [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity". Valence relates to the positivity (or negativity) of the song - so either positive or negative emotions.<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
The most important Spotify features that we will use in our technology are "tempo", "danceability", "energy" and "valence". Tempo is just the BPM of the song as defined by existing DSP algorithms. It is a floating point number that has no limit. [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] describes danceability as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity." This is a floating point number with a range from 0 (not danceable) to 1 (very danceable). Energy is described by Spotify as the "perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy." It is a floating point number with a range from 0 (no energy) to 1 (high energy). Valence represents the degree of positiveness of a song. It is a floating point number that ranges from 0 (very negative) to 1 (very positive). <br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section and in the [[#The Effenaar|conversation with the Effenaar]]. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition but there is a difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching will slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. <span style="background:yellow">We decided to focus on engineering a proper pre-filter</span> and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or equation was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no equation can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs.<br />
<br />
The word excitement is very much linked to a highly energized state of positive feelings. As it is given a definition by the [https://www.oxfordlearnersdictionaries.com/definition/english/excitement Oxford dictionary]: "the state of feeling or showing happiness and enthusiasm." That is why we chose the features "energy" and "valence" to predict this variable with.<br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction equation for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please note that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the equation for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno is the only one which turned out significant, R² = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate equation for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R² = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R² = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R² = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear equation to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This equation is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the equation, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction equation for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system will be used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust equation for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
One module of the system is the excitement matching algorithm. What we mean by this is a system that takes in a graph (which can be a set of points) from the user with the desired course of "excitement" of the music set. This is called a Qualitative Excitement Trajectory (QET). Thus, we want a module that takes in a QET from the user and transforms it into a mathematical equation, because a graph is what humans can work with, while an equation is what computers can work with. With this mathematical equation the module can output different values of "excitement" for every point in time in the music set and thus also outputs "excitement" values at points where a new music track should start. These latter points can then be matched with existing music tracks with an "excitement" value close to this desired value by a Spotify filter. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes long music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at an excitement of 0.7, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical equation for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical equation for the QET. This equation can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled equation for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to put in his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b><span style="background:yellow">UNDER CONSTRUCTION?</span></b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together seamlessly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and the Spotify features "tempo", "danceability", "energy" and "valence". This input is made possible by the [[#User interface|user interface]]. In that sense, the pre-filter puts out an array of tracks that is already filtered according to the user's desire. Next, the filtered tracks will be matched according the desired [[#Algorithms for track selection|QET]] <span style="background:yellow">(we will use an excitement graph instead of a tempo graph but the principle is the same).</span> The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Validation of the user needs=<br />
<br />
==Primary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| The DJ-robot is more valued than a human performer.<br />
| <br />
*Blind testen naast een menselijke DJ, als zijnde een turing test<br />
**Cafe/disco, De ene avond een menselijke DJ laten spelen en een andere avond de robot, maar laten lijken alsof een mens de plaatjes draait.<br />
**Gasten de muziekbeleving laten beoordelen. Gemiddeld aantal gasten per uur tellen.<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Silent disco met meerdere kanalen, 1 met de robot-DJ, 1 met echte DJ (wederom blind)<br />
**Om de zoveel tijd tellen welke kanalen beluisterd worden<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Een groep DJ’s een muziek set laten maken. De robot eenzelfde set laten maken. Dus zelfde genre, zelfde QET etc.<br />
**Een groep willekeurige mensen de sets rangschikken van goed naar slecht.<br />
|-<br />
| The system's user interface is easy to understand, no experts needed.<br />
| <br />
*Usability testing<br />
**…<br />
|-<br />
|}<br />
<br />
==Secondary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| <br />
*The music set played is structured and progressive.<br />
| <br />
*…<br />
|-<br />
|<br />
*Track selection should fit the audience background.<br />
**The system selects appropriate tracks regarding genre.<br />
**The musical presentation should reflect the audience energy level.<br />
| <br />
*…<br />
|-<br />
| <br />
*There is a balance between playing familiar tracks and providing rare, new music.<br />
**The system selects popular tracks that are valued by the audience.<br />
**Similar tracks to what the audience is into are played.<br />
| <br />
*…<br />
|-<br />
| <br />
*The dancers are taken on a cohesive and dynamical music journey.<br />
| <br />
*…<br />
|-<br />
| <br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
| <br />
*Not in the scope of our project.<br />
|-<br />
|}<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
[[Weekly Log PRE2019 3 Group15]]<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| Backend development + CSS slider development <i>7 hours</i><br />
| Did research on MATLAB Compiler for Java application: <i>3 hours</i><br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| Learned CSS + started integration of backend <i>6 hours</i><br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Updated the API reqiest code: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Tried to transform MATLAB functions to Java: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| Worked on a [[#A GUI for QET matching|GUI]] for QET matching: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| Worked on the UI <i>6 hours</i><br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| Worked on [[#A GUI for QET matching|GUI]] for QET matching: <i>3 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| Reported my progression on the wiki and editing existing pages: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>1.5 hour</i><br/>Starting on validation of user needs: <i>1.5 hour</i><br />
| Worked on the UI <i>5 hours</i><br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=90132PRE2019 3 Group152020-03-31T10:02:22Z<p>S165296: /* Feedforward */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Introduction=<br />
==Problem Statement and Objectives==<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
==Approach, Milestones, and Deliverables==<br />
===Approach===<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
===<span style="background:yellow">Milestones</span>===<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
===Deliverables===<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
Danceability relates to the extent to which it is easy to dance on the music. This is a feature, described by [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity". Valence relates to the positivity (or negativity) of the song - so either positive or negative emotions.<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
The most important Spotify features that we will use in our technology are "tempo", "danceability", "energy" and "valence". Tempo is just the BPM of the song as defined by existing DSP algorithms. It is a floating point number that has no limit. [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] describes danceability as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity." This is a floating point number with a range from 0 (not danceable) to 1 (very danceable). Energy is described by Spotify as the "perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy." It is a floating point number with a range from 0 (no energy) to 1 (high energy). Valence represents the degree of positiveness of a song. It is a floating point number that ranges from 0 (very negative) to 1 (very positive). <br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section and in the [[#The Effenaar|conversation with the Effenaar]]. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition but there is a difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching will slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. <span style="background:yellow">We decided to focus on engineering a proper pre-filter</span> and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or equation was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no equation can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs.<br />
<br />
The word excitement is very much linked to a highly energized state of positive feelings. As it is given a definition by the [https://www.oxfordlearnersdictionaries.com/definition/english/excitement Oxford dictionary]: "the state of feeling or showing happiness and enthusiasm." That is why we chose the features "energy" and "valence" to predict this variable with.<br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction equation for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please note that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the equation for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno is the only one which turned out significant, R² = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate equation for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R² = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R² = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R² = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear equation to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This equation is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the equation, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction equation for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system will be used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust equation for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
One module of the system is the excitement matching algorithm. What we mean by this is a system that takes in a graph (which can be a set of points) from the user with the desired course of "excitement" of the music set. This is called a Qualitative Excitement Trajectory (QET). Thus, we want a module that takes in a QET from the user and transforms it into a mathematical equation, because a graph is what humans can work with, while an equation is what computers can work with. With this mathematical equation the module can output different values of "excitement" for every point in time in the music set and thus also outputs "excitement" values at points where a new music track should start. These latter points can then be matched with existing music tracks with an "excitement" value close to this desired value by a Spotify filter. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes long music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at an excitement of 0.7, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical equation for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical equation for the QET. This equation can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled equation for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to put in his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b><span style="background:yellow">UNDER CONSTRUCTION?</span></b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together seamlessly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and <span style="background:yellow">SPOTIFY FEATURES (NOG BESLISSEN WELKE)</span> based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter puts out an array of tracks that is already filtered according to the user's desire. Next, the filtered tracks will be matched according the desired [[#Algorithms for track selection|QET]] <span style="background:yellow">(we will use an excitement graph instead of a tempo graph but the principle is the same).</span> The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Validation of the user needs=<br />
<br />
==Primary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| The DJ-robot is more valued than a human performer.<br />
| <br />
*Blind testen naast een menselijke DJ, als zijnde een turing test<br />
**Cafe/disco, De ene avond een menselijke DJ laten spelen en een andere avond de robot, maar laten lijken alsof een mens de plaatjes draait.<br />
**Gasten de muziekbeleving laten beoordelen. Gemiddeld aantal gasten per uur tellen.<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Silent disco met meerdere kanalen, 1 met de robot-DJ, 1 met echte DJ (wederom blind)<br />
**Om de zoveel tijd tellen welke kanalen beluisterd worden<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Een groep DJ’s een muziek set laten maken. De robot eenzelfde set laten maken. Dus zelfde genre, zelfde QET etc.<br />
**Een groep willekeurige mensen de sets rangschikken van goed naar slecht.<br />
|-<br />
| The system's user interface is easy to understand, no experts needed.<br />
| <br />
*Usability testing<br />
**…<br />
|-<br />
|}<br />
<br />
==Secondary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| <br />
*The music set played is structured and progressive.<br />
| <br />
*…<br />
|-<br />
|<br />
*Track selection should fit the audience background.<br />
**The system selects appropriate tracks regarding genre.<br />
**The musical presentation should reflect the audience energy level.<br />
| <br />
*…<br />
|-<br />
| <br />
*There is a balance between playing familiar tracks and providing rare, new music.<br />
**The system selects popular tracks that are valued by the audience.<br />
**Similar tracks to what the audience is into are played.<br />
| <br />
*…<br />
|-<br />
| <br />
*The dancers are taken on a cohesive and dynamical music journey.<br />
| <br />
*…<br />
|-<br />
| <br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
| <br />
*Not in the scope of our project.<br />
|-<br />
|}<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
[[Weekly Log PRE2019 3 Group15]]<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| Backend development + CSS slider development <i>7 hours</i><br />
| Did research on MATLAB Compiler for Java application: <i>3 hours</i><br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| Learned CSS + started integration of backend <i>6 hours</i><br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Updated the API reqiest code: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Tried to transform MATLAB functions to Java: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| Worked on a [[#A GUI for QET matching|GUI]] for QET matching: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| Worked on the UI <i>6 hours</i><br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| Worked on [[#A GUI for QET matching|GUI]] for QET matching: <i>3 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| Reported my progression on the wiki and editing existing pages: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>1.5 hour</i><br/>Starting on validation of user needs: <i>1.5 hour</i><br />
| Worked on the UI <i>5 hours</i><br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=File:System_overview_graphic.JPG&diff=90110File:System overview graphic.JPG2020-03-31T09:48:42Z<p>S165296: uploaded a new version of "File:System overview graphic.JPG"</p>
<hr />
<div></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=90106PRE2019 3 Group152020-03-31T09:39:59Z<p>S165296: /* Excitement matching with QET */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Introduction=<br />
==Problem Statement and Objectives==<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
==Approach, Milestones, and Deliverables==<br />
===Approach===<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
===<span style="background:yellow">Milestones</span>===<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
===Deliverables===<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
Danceability relates to the extent to which it is easy to dance on the music. This is a feature, described by [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity". Valence relates to the positivity (or negativity) of the song - so either positive or negative emotions.<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section and in the [[#The Effenaar|conversation with the Effenaar]]. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition but there is a difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching will slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. <span style="background:yellow">We decided to focus on engineering a proper pre-filter</span> and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or equation was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no equation can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs.<br />
<br />
The word excitement is very much linked to a highly energized state of positive feelings. As it is given a definition by the [https://www.oxfordlearnersdictionaries.com/definition/english/excitement Oxford dictionary]: "the state of feeling or showing happiness and enthusiasm." That is why we chose the features "energy" and "valence" to predict this variable with.<br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction equation for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please note that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the equation for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno is the only one which turned out significant, R² = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate equation for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R² = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R² = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R² = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear equation to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This equation is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the equation, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction equation for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system will be used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust equation for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
One module of the system is the excitement matching algorithm. What we mean by this is a system that takes in a graph (which can be a set of points) from the user with the desired course of "excitement" of the music set. This is called a Qualitative Excitement Trajectory (QET). Thus, we want a module that takes in a QET from the user and transforms it into a mathematical equation, because a graph is what humans can work with, while an equation is what computers can work with. With this mathematical equation the module can output different values of "excitement" for every point in time in the music set and thus also outputs "excitement" values at points where a new music track should start. These latter points can then be matched with existing music tracks with an "excitement" value close to this desired value by a Spotify filter. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes long music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at an excitement of 0.7, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical equation for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical equation for the QET. This equation can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled equation for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to put in his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b><span style="background:yellow">UNDER CONSTRUCTION?</span></b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together seamlessly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and <span style="background:yellow">SPOTIFY FEATURES (NOG BESLISSEN WELKE)</span> based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter puts out an array of tracks that is already filtered according to the user's desire. Next, the filtered tracks will be matched according the desired [[#Algorithms for track selection|QET]] <span style="background:yellow">(we will use an excitement graph instead of a tempo graph but the principle is the same).</span> The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Validation of the user needs=<br />
<br />
==Primary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| The DJ-robot is more valued than a human performer.<br />
| <br />
*Blind testen naast een menselijke DJ, als zijnde een turing test<br />
**Cafe/disco, De ene avond een menselijke DJ laten spelen en een andere avond de robot, maar laten lijken alsof een mens de plaatjes draait.<br />
**Gasten de muziekbeleving laten beoordelen. Gemiddeld aantal gasten per uur tellen.<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Silent disco met meerdere kanalen, 1 met de robot-DJ, 1 met echte DJ (wederom blind)<br />
**Om de zoveel tijd tellen welke kanalen beluisterd worden<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Een groep DJ’s een muziek set laten maken. De robot eenzelfde set laten maken. Dus zelfde genre, zelfde QET etc.<br />
**Een groep willekeurige mensen de sets rangschikken van goed naar slecht.<br />
|-<br />
| The system's user interface is easy to understand, no experts needed.<br />
| <br />
*Usability testing<br />
**…<br />
|-<br />
|}<br />
<br />
==Secondary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| <br />
*The music set played is structured and progressive.<br />
| <br />
*…<br />
|-<br />
|<br />
*Track selection should fit the audience background.<br />
**The system selects appropriate tracks regarding genre.<br />
**The musical presentation should reflect the audience energy level.<br />
| <br />
*…<br />
|-<br />
| <br />
*There is a balance between playing familiar tracks and providing rare, new music.<br />
**The system selects popular tracks that are valued by the audience.<br />
**Similar tracks to what the audience is into are played.<br />
| <br />
*…<br />
|-<br />
| <br />
*The dancers are taken on a cohesive and dynamical music journey.<br />
| <br />
*…<br />
|-<br />
| <br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
| <br />
*Not in the scope of our project.<br />
|-<br />
|}<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| Backend development + CSS slider development <i>7 hours</i><br />
| Did research on MATLAB Compiler for Java application: <i>3 hours</i><br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| Learned CSS + started integration of backend <i>6 hours</i><br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Updated the API reqiest code: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Tried to transform MATLAB functions to Java: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| Worked on a [[#A GUI for QET matching|GUI]] for QET matching: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| Worked on the UI <i>6 hours</i><br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| Worked on [[#A GUI for QET matching|GUI]] for QET matching: <i>3 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| Reported my progression on the wiki and editing existing pages: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>1.5 hour</i><br/>Starting on validation of user needs: <i>1.5 hour</i><br />
| Worked on the UI <i>5 hours</i><br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=90103PRE2019 3 Group152020-03-31T09:38:40Z<p>S165296: /* Excitement matching with QET */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Introduction=<br />
==Problem Statement and Objectives==<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
==Approach, Milestones, and Deliverables==<br />
===Approach===<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
===<span style="background:yellow">Milestones</span>===<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
===Deliverables===<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
Danceability relates to the extent to which it is easy to dance on the music. This is a feature, described by [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity". Valence relates to the positivity (or negativity) of the song - so either positive or negative emotions.<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section and in the [[#The Effenaar|conversation with the Effenaar]]. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition but there is a difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching will slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. <span style="background:yellow">We decided to focus on engineering a proper pre-filter</span> and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or equation was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no equation can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs.<br />
<br />
The word excitement is very much linked to a highly energized state of positive feelings. As it is given a definition by the [https://www.oxfordlearnersdictionaries.com/definition/english/excitement Oxford dictionary]: "the state of feeling or showing happiness and enthusiasm." That is why we chose the features "energy" and "valence" to predict this variable with.<br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction equation for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please note that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the equation for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno is the only one which turned out significant, R² = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate equation for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R² = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R² = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R² = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear equation to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This equation is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the equation, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction equation for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system will be used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust equation for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
One module of the system is the excitement matching algorithm. What we mean by this is a system that takes in a graph (which can be a set of points) from the user with the desired course of "excitement" of the music set. This is called a Qualitative Excitement Trajectory (QET). Thus, we want a module that takes in a QET from the user and transforms it into a mathematical equation, because a graph is what humans can work with, while an equation is what computers can work with. With this mathematical equation the module can output different values of "excitement" for every point in time in the music set and thus also outputs "excitement" values at points where a new music track should start. These latter points can then be matched with existing music tracks with an "excitement" value close to this desired value. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes long music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at an excitement of 0.7, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical equation for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical equation for the QET. This equation can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled equation for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to put in his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b><span style="background:yellow">UNDER CONSTRUCTION?</span></b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together seamlessly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and <span style="background:yellow">SPOTIFY FEATURES (NOG BESLISSEN WELKE)</span> based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter puts out an array of tracks that is already filtered according to the user's desire. Next, the filtered tracks will be matched according the desired [[#Algorithms for track selection|QET]] <span style="background:yellow">(we will use an excitement graph instead of a tempo graph but the principle is the same).</span> The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Validation of the user needs=<br />
<br />
==Primary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| The DJ-robot is more valued than a human performer.<br />
| <br />
*Blind testen naast een menselijke DJ, als zijnde een turing test<br />
**Cafe/disco, De ene avond een menselijke DJ laten spelen en een andere avond de robot, maar laten lijken alsof een mens de plaatjes draait.<br />
**Gasten de muziekbeleving laten beoordelen. Gemiddeld aantal gasten per uur tellen.<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Silent disco met meerdere kanalen, 1 met de robot-DJ, 1 met echte DJ (wederom blind)<br />
**Om de zoveel tijd tellen welke kanalen beluisterd worden<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Een groep DJ’s een muziek set laten maken. De robot eenzelfde set laten maken. Dus zelfde genre, zelfde QET etc.<br />
**Een groep willekeurige mensen de sets rangschikken van goed naar slecht.<br />
|-<br />
| The system's user interface is easy to understand, no experts needed.<br />
| <br />
*Usability testing<br />
**…<br />
|-<br />
|}<br />
<br />
==Secondary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| <br />
*The music set played is structured and progressive.<br />
| <br />
*…<br />
|-<br />
|<br />
*Track selection should fit the audience background.<br />
**The system selects appropriate tracks regarding genre.<br />
**The musical presentation should reflect the audience energy level.<br />
| <br />
*…<br />
|-<br />
| <br />
*There is a balance between playing familiar tracks and providing rare, new music.<br />
**The system selects popular tracks that are valued by the audience.<br />
**Similar tracks to what the audience is into are played.<br />
| <br />
*…<br />
|-<br />
| <br />
*The dancers are taken on a cohesive and dynamical music journey.<br />
| <br />
*…<br />
|-<br />
| <br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
| <br />
*Not in the scope of our project.<br />
|-<br />
|}<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| Backend development + CSS slider development <i>7 hours</i><br />
| Did research on MATLAB Compiler for Java application: <i>3 hours</i><br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| Learned CSS + started integration of backend <i>6 hours</i><br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Updated the API reqiest code: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Tried to transform MATLAB functions to Java: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| Worked on a [[#A GUI for QET matching|GUI]] for QET matching: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| Worked on the UI <i>6 hours</i><br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| Worked on [[#A GUI for QET matching|GUI]] for QET matching: <i>3 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| Reported my progression on the wiki and editing existing pages: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>1.5 hour</i><br/>Starting on validation of user needs: <i>1.5 hour</i><br />
| Worked on the UI <i>5 hours</i><br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=90101PRE2019 3 Group152020-03-31T09:36:19Z<p>S165296: /* Excitement matching with QET */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Introduction=<br />
==Problem Statement and Objectives==<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
==Approach, Milestones, and Deliverables==<br />
===Approach===<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
===<span style="background:yellow">Milestones</span>===<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
===Deliverables===<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
Danceability relates to the extent to which it is easy to dance on the music. This is a feature, described by [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity". Valence relates to the positivity (or negativity) of the song - so either positive or negative emotions.<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section and in the [[#The Effenaar|conversation with the Effenaar]]. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition but there is a difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching will slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. <span style="background:yellow">We decided to focus on engineering a proper pre-filter</span> and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or equation was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no equation can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs.<br />
<br />
The word excitement is very much linked to a highly energized state of positive feelings. As it is given a definition by the [https://www.oxfordlearnersdictionaries.com/definition/english/excitement Oxford dictionary]: "the state of feeling or showing happiness and enthusiasm." That is why we chose the features "energy" and "valence" to predict this variable with.<br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction equation for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please note that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the equation for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno is the only one which turned out significant, R² = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate equation for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R² = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R² = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R² = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear equation to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This equation is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the equation, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction equation for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system will be used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust equation for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
One module of the system is the excitement matching algorithm. What we mean by this is a system that takes in a graph or a set of points from the user with the desired course of "excitement" of the music set. This is called a Qualitative Excitement Trajectory (QET). Thus, we want a module that takes in a QET from the user and transforms it into a mathematical equation, because a graph is what humans can work with, while a equation is what computers can work with. With this mathematical equation the module can output different values of "excitement" for every point in time in the music set and thus also outputs "excitement" values at points where a new music track should start. These latter points can then be matched with existing music tracks with an "excitement" value close to this desired value. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes long music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at an excitement of 0.7, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical equation for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical equation for the QET. This equation can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled equation for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to put in his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b><span style="background:yellow">UNDER CONSTRUCTION?</span></b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together seamlessly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and <span style="background:yellow">SPOTIFY FEATURES (NOG BESLISSEN WELKE)</span> based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter puts out an array of tracks that is already filtered according to the user's desire. Next, the filtered tracks will be matched according the desired [[#Algorithms for track selection|QET]] <span style="background:yellow">(we will use an excitement graph instead of a tempo graph but the principle is the same).</span> The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Validation of the user needs=<br />
<br />
==Primary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| The DJ-robot is more valued than a human performer.<br />
| <br />
*Blind testen naast een menselijke DJ, als zijnde een turing test<br />
**Cafe/disco, De ene avond een menselijke DJ laten spelen en een andere avond de robot, maar laten lijken alsof een mens de plaatjes draait.<br />
**Gasten de muziekbeleving laten beoordelen. Gemiddeld aantal gasten per uur tellen.<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Silent disco met meerdere kanalen, 1 met de robot-DJ, 1 met echte DJ (wederom blind)<br />
**Om de zoveel tijd tellen welke kanalen beluisterd worden<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Een groep DJ’s een muziek set laten maken. De robot eenzelfde set laten maken. Dus zelfde genre, zelfde QET etc.<br />
**Een groep willekeurige mensen de sets rangschikken van goed naar slecht.<br />
|-<br />
| The system's user interface is easy to understand, no experts needed.<br />
| <br />
*Usability testing<br />
**…<br />
|-<br />
|}<br />
<br />
==Secondary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| <br />
*The music set played is structured and progressive.<br />
| <br />
*…<br />
|-<br />
|<br />
*Track selection should fit the audience background.<br />
**The system selects appropriate tracks regarding genre.<br />
**The musical presentation should reflect the audience energy level.<br />
| <br />
*…<br />
|-<br />
| <br />
*There is a balance between playing familiar tracks and providing rare, new music.<br />
**The system selects popular tracks that are valued by the audience.<br />
**Similar tracks to what the audience is into are played.<br />
| <br />
*…<br />
|-<br />
| <br />
*The dancers are taken on a cohesive and dynamical music journey.<br />
| <br />
*…<br />
|-<br />
| <br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
| <br />
*Not in the scope of our project.<br />
|-<br />
|}<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| Backend development + CSS slider development <i>7 hours</i><br />
| Did research on MATLAB Compiler for Java application: <i>3 hours</i><br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| Learned CSS + started integration of backend <i>6 hours</i><br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Updated the API reqiest code: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Tried to transform MATLAB functions to Java: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| Worked on a [[#A GUI for QET matching|GUI]] for QET matching: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| Worked on the UI <i>6 hours</i><br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| Worked on [[#A GUI for QET matching|GUI]] for QET matching: <i>3 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| Reported my progression on the wiki and editing existing pages: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>1.5 hour</i><br/>Starting on validation of user needs: <i>1.5 hour</i><br />
| Worked on the UI <i>5 hours</i><br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=90073PRE2019 3 Group152020-03-31T09:21:23Z<p>S165296: /* Excitement prediction with multiple regression */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==<span style="background:yellow">Milestones</span>==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
Danceability relates to the extent to which it is easy to dance on the music. This is a feature, described by [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity". Valence relates to the positivity (or negativity) of the song - so either positive or negative emotions.<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section and in the [[#The Effenaar|conversation with the Effenaar]]. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition but there is a difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching will slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. <span style="background:yellow">We decided to focus on engineering a proper pre-filter</span> and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or equation was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no equation can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs.<br />
<br />
The word excitement is very much linked to a highly energized state of positive feelings. As it is given a definition by the [https://www.oxfordlearnersdictionaries.com/definition/english/excitement Oxford dictionary]: "the state of feeling or showing happiness and enthusiasm." That is why we chose the features "energy" and "valence" to predict this variable with.<br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction equation for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please note that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the equation for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno is the only one which turned out significant, R² = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate equation for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R² = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R² = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R² = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear equation to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This equation is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the equation, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction equation for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system will be used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust equation for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the <span style="background:yellow">excitement matching algorithm</span> we want a module that takes in a QET from the user and transforms it into a mathematical equation, because a graph is what humans can work with, while a equation is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes long music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at an excitement of 0.7, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical equation for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical equation for the QET. This equation can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled equation for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to put in his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b><span style="background:yellow">UNDER CONSTRUCTION?</span></b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together seamlessly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and <span style="background:yellow">SPOTIFY FEATURES (NOG BESLISSEN WELKE)</span> based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter puts out an array of tracks that is already filtered according to the user's desire. Next, the filtered tracks will be matched according the desired [[#Algorithms for track selection|QET]] <span style="background:yellow">(we will use an excitement graph instead of a tempo graph but the principle is the same).</span> The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Validation of the user needs=<br />
<br />
==Primary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| The DJ-robot is more valued than a human performer.<br />
| <br />
*Blind testen naast een menselijke DJ, als zijnde een turing test<br />
**Cafe/disco, De ene avond een menselijke DJ laten spelen en een andere avond de robot, maar laten lijken alsof een mens de plaatjes draait.<br />
**Gasten de muziekbeleving laten beoordelen. Gemiddeld aantal gasten per uur tellen.<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Silent disco met meerdere kanalen, 1 met de robot-DJ, 1 met echte DJ (wederom blind)<br />
**Om de zoveel tijd tellen welke kanalen beluisterd worden<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Een groep DJ’s een muziek set laten maken. De robot eenzelfde set laten maken. Dus zelfde genre, zelfde QET etc.<br />
**Een groep willekeurige mensen de sets rangschikken van goed naar slecht.<br />
|-<br />
| The system's user interface is easy to understand, no experts needed.<br />
| <br />
*Usability testing<br />
**…<br />
|-<br />
|}<br />
<br />
==Secondary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| <br />
*The music set played is structured and progressive.<br />
| <br />
*…<br />
|-<br />
|<br />
*Track selection should fit the audience background.<br />
**The system selects appropriate tracks regarding genre.<br />
**The musical presentation should reflect the audience energy level.<br />
| <br />
*…<br />
|-<br />
| <br />
*There is a balance between playing familiar tracks and providing rare, new music.<br />
**The system selects popular tracks that are valued by the audience.<br />
**Similar tracks to what the audience is into are played.<br />
| <br />
*…<br />
|-<br />
| <br />
*The dancers are taken on a cohesive and dynamical music journey.<br />
| <br />
*…<br />
|-<br />
| <br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
| <br />
*Not in the scope of our project.<br />
|-<br />
|}<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| Backend development + CSS slider development <i>7 hours</i><br />
| Did research on MATLAB Compiler for Java application: <i>3 hours</i><br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| Learned CSS + started integration of backend <i>6 hours</i><br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Updated the API reqiest code: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Tried to transform MATLAB functions to Java: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| Worked on a [[#A GUI for QET matching|GUI]] for QET matching: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| Worked on the UI <i>6 hours</i><br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| Worked on [[#A GUI for QET matching|GUI]] for QET matching: <i>3 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| Reported my progression on the wiki and editing existing pages: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>1.5 hour</i><br/>Starting on validation of user needs: <i>1.5 hour</i><br />
| Worked on the UI <i>5 hours</i><br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=90065PRE2019 3 Group152020-03-31T09:09:04Z<p>S165296: /* Excitement prediction with multiple regression */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==<span style="background:yellow">Milestones</span>==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
Danceability relates to the extent to which it is easy to dance on the music. This is a feature, described by [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity". Valence relates to the positivity (or negativity) of the song - so either positive or negative emotions.<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section and in the [[#The Effenaar|conversation with the Effenaar]]. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition but there is a difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching will slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. <span style="background:yellow">We decided to focus on engineering a proper pre-filter</span> and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or equation was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no equation can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction equation for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please note that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the equation for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno is the only one which turned out significant, R² = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate equation for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R² = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R² = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R² = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear equation to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This equation is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the equation, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction equation for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system will be used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust equation for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the <span style="background:yellow">excitement matching algorithm</span> we want a module that takes in a QET from the user and transforms it into a mathematical equation, because a graph is what humans can work with, while a equation is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes long music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at an excitement of 0.7, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical equation for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical equation for the QET. This equation can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled equation for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to put in his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b><span style="background:yellow">UNDER CONSTRUCTION?</span></b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together seamlessly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and <span style="background:yellow">SPOTIFY FEATURES (NOG BESLISSEN WELKE)</span> based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter puts out an array of tracks that is already filtered according to the user's desire. Next, the filtered tracks will be matched according the desired [[#Algorithms for track selection|QET]] <span style="background:yellow">(we will use an excitement graph instead of a tempo graph but the principle is the same).</span> The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Validation of the user needs=<br />
<br />
==Primary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| The DJ-robot is more valued than a human performer.<br />
| <br />
*Blind testen naast een menselijke DJ, als zijnde een turing test<br />
**Cafe/disco, De ene avond een menselijke DJ laten spelen en een andere avond de robot, maar laten lijken alsof een mens de plaatjes draait.<br />
**Gasten de muziekbeleving laten beoordelen. Gemiddeld aantal gasten per uur tellen.<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Silent disco met meerdere kanalen, 1 met de robot-DJ, 1 met echte DJ (wederom blind)<br />
**Om de zoveel tijd tellen welke kanalen beluisterd worden<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Een groep DJ’s een muziek set laten maken. De robot eenzelfde set laten maken. Dus zelfde genre, zelfde QET etc.<br />
**Een groep willekeurige mensen de sets rangschikken van goed naar slecht.<br />
|-<br />
| The system's user interface is easy to understand, no experts needed.<br />
| <br />
*Usability testing<br />
**…<br />
|-<br />
|}<br />
<br />
==Secondary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| <br />
*The music set played is structured and progressive.<br />
| <br />
*…<br />
|-<br />
|<br />
*Track selection should fit the audience background.<br />
**The system selects appropriate tracks regarding genre.<br />
**The musical presentation should reflect the audience energy level.<br />
| <br />
*…<br />
|-<br />
| <br />
*There is a balance between playing familiar tracks and providing rare, new music.<br />
**The system selects popular tracks that are valued by the audience.<br />
**Similar tracks to what the audience is into are played.<br />
| <br />
*…<br />
|-<br />
| <br />
*The dancers are taken on a cohesive and dynamical music journey.<br />
| <br />
*…<br />
|-<br />
| <br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
| <br />
*Not in the scope of our project.<br />
|-<br />
|}<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| Backend development + CSS slider development <i>7 hours</i><br />
| Did research on MATLAB Compiler for Java application: <i>3 hours</i><br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| Learned CSS + started integration of backend <i>6 hours</i><br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Updated the API reqiest code: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Tried to transform MATLAB functions to Java: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| Worked on a [[#A GUI for QET matching|GUI]] for QET matching: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| Worked on the UI <i>6 hours</i><br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| Worked on [[#A GUI for QET matching|GUI]] for QET matching: <i>3 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| Reported my progression on the wiki and editing existing pages: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>1.5 hour</i><br/>Starting on validation of user needs: <i>1.5 hour</i><br />
| Worked on the UI <i>5 hours</i><br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=90060PRE2019 3 Group152020-03-31T09:01:29Z<p>S165296: /* Feedback */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==<span style="background:yellow">Milestones</span>==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
Danceability relates to the extent to which it is easy to dance on the music. This is a feature, described by [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity". Valence relates to the positivity (or negativity) of the song - so either positive or negative emotions.<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section and in the [[#The Effenaar|conversation with the Effenaar]]. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition but there is a difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching will slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. <span style="background:yellow">We decided to focus on engineering a proper pre-filter</span> and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or equation was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no equation can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction equation for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the equation for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno is the only one which turned out significant, R² = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate equation for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R² = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R² = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting equation is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R² = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear equation to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This equation is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the equation, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction equation for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system will be used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust equation for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the <span style="background:yellow">excitement matching algorithm</span> we want a module that takes in a QET from the user and transforms it into a mathematical equation, because a graph is what humans can work with, while a equation is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes long music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at an excitement of 0.7, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical equation for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical equation for the QET. This equation can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled equation for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to put in his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b><span style="background:yellow">UNDER CONSTRUCTION?</span></b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together seamlessly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and <span style="background:yellow">SPOTIFY FEATURES (NOG BESLISSEN WELKE)</span> based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter puts out an array of tracks that is already filtered according to the user's desire. Next, the filtered tracks will be matched according the desired [[#Algorithms for track selection|QET]] <span style="background:yellow">(we will use an excitement graph instead of a tempo graph but the principle is the same).</span> The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Validation of the user needs=<br />
<br />
==Primary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| The DJ-robot is more valued than a human performer.<br />
| <br />
*Blind testen naast een menselijke DJ, als zijnde een turing test<br />
**Cafe/disco, De ene avond een menselijke DJ laten spelen en een andere avond de robot, maar laten lijken alsof een mens de plaatjes draait.<br />
**Gasten de muziekbeleving laten beoordelen. Gemiddeld aantal gasten per uur tellen.<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Silent disco met meerdere kanalen, 1 met de robot-DJ, 1 met echte DJ (wederom blind)<br />
**Om de zoveel tijd tellen welke kanalen beluisterd worden<br />
**Dit meerdere avonden doen om statistisch significant resultaat te krijgen<br />
<br />
*Een groep DJ’s een muziek set laten maken. De robot eenzelfde set laten maken. Dus zelfde genre, zelfde QET etc.<br />
**Een groep willekeurige mensen de sets rangschikken van goed naar slecht.<br />
|-<br />
| The system's user interface is easy to understand, no experts needed.<br />
| <br />
*Usability testing<br />
**…<br />
|-<br />
|}<br />
<br />
==Secondary user needs==<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| User need<br />
! scope="col"| Validation method(s)<br />
|-<br />
| <br />
*The music set played is structured and progressive.<br />
| <br />
*…<br />
|-<br />
|<br />
*Track selection should fit the audience background.<br />
**The system selects appropriate tracks regarding genre.<br />
**The musical presentation should reflect the audience energy level.<br />
| <br />
*…<br />
|-<br />
| <br />
*There is a balance between playing familiar tracks and providing rare, new music.<br />
**The system selects popular tracks that are valued by the audience.<br />
**Similar tracks to what the audience is into are played.<br />
| <br />
*…<br />
|-<br />
| <br />
*The dancers are taken on a cohesive and dynamical music journey.<br />
| <br />
*…<br />
|-<br />
| <br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
| <br />
*Not in the scope of our project.<br />
|-<br />
|}<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| Backend development + CSS slider development <i>7 hours</i><br />
| Did research on MATLAB Compiler for Java application: <i>3 hours</i><br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| Learned CSS + started integration of backend <i>6 hours</i><br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Updated the API reqiest code: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Tried to transform MATLAB functions to Java: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| Worked on a [[#A GUI for QET matching|GUI]] for QET matching: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| Worked on the UI <i>6 hours</i><br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| Worked on [[#A GUI for QET matching|GUI]] for QET matching: <i>3 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| Reported my progression on the wiki and editing existing pages: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>1.5 hour</i><br/>Starting on validation of user needs: <i>1.5 hour</i><br />
| Worked on the UI <i>5 hours</i><br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89471PRE2019 3 Group152020-03-29T15:24:07Z<p>S165296: /* Weekly Log */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
Danceability relates to the extent to which it is easy to dance on the music. This is a feature, described by [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity". Valence relates to the positivity (or negativity) of the song - so either positive or negative emotions.<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| Did research on MATLAB Compiler for Java application: <i>3 hours</i><br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br/>Tried to transform MATLAB functions to Java: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| Worked on a [[#A GUI for QET matching|GUI]] for QET matching: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| Worked on [[#A GUI for QET matching|GUI]] for QET matching: <i>3 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| Reported my progression on the wiki and editing existing pages: <i>2 hours</i><br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89470PRE2019 3 Group152020-03-29T15:22:36Z<p>S165296: /* Weekly Log */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
Danceability relates to the extent to which it is easy to dance on the music. This is a feature, described by [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity". Valence relates to the positivity (or negativity) of the song - so either positive or negative emotions.<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| Did research on MATLAB Compiler for Java application: <i>3 hours</i><br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i>, Tried to transform MATLAB fucntions to Java: <i>3 hours</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| Worked on a [[#A GUI for QET matching|GUI]] for QET matching: <i>2 hours</i><br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| Worked on [[#A GUI for QET matching|GUI]] for QET matching: <i>3 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| Reported my progression on the wiki and editing existing pages: <i>2 hours</i><br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89464PRE2019 3 Group152020-03-29T15:13:39Z<p>S165296: /* Defining and influencing characteristics of the music */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
Danceability relates to the extent to which it is easy to dance on the music. This is a feature, described by [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] as "to what extent can people dance on it based on tempo, rhythm stability, beat strength, and overall regularity". Valence relates to the positivity (or negativity) of the song - so either positive or negative emotions.<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| <br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| <br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89461PRE2019 3 Group152020-03-29T15:08:29Z<p>S165296: /* Defining and influencing characteristics of the music */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy, tempo, danceability and valence.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| <br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| <br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89340PRE2019 3 Group152020-03-29T10:35:40Z<p>S165296: /* The Effenaar */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 The Effenaar has developed into one of the biggest music venues of the Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focuses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However, retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover, they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are <span style="background:yellow">chaos, energy and tempo</span> <b><span style="background:yellow">ADD HERE MORE FEATURES DURING THE PROJECT</span></b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| <br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| <br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89338PRE2019 3 Group152020-03-29T10:31:00Z<p>S165296: /* A GUI for QET matching */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 has The Effenaar developed into one of the biggest music venues of The Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focusses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are <span style="background:yellow">chaos, energy and tempo</span> <b><span style="background:yellow">ADD HERE MORE FEATURES DURING THE PROJECT</span></b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for deciding on a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| <br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| <br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89337PRE2019 3 Group152020-03-29T10:30:40Z<p>S165296: /* A GUI for QET matching */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 has The Effenaar developed into one of the biggest music venues of The Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focusses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are <span style="background:yellow">chaos, energy and tempo</span> <b><span style="background:yellow">ADD HERE MORE FEATURES DURING THE PROJECT</span></b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
By providing the user a visual tool for providing a QET, the system answers to the primary user need of an easy to understand user interface for which no experts are needed to operate it. Besides, it answers to the secondary user needs of a structured and progressive music set and that dancers are taken on a cohesive and dynamical music journey.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| <br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| <br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89335PRE2019 3 Group152020-03-29T10:22:18Z<p>S165296: /* A GUI for QET matching */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 has The Effenaar developed into one of the biggest music venues of The Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focusses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are <span style="background:yellow">chaos, energy and tempo</span> <b><span style="background:yellow">ADD HERE MORE FEATURES DURING THE PROJECT</span></b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described and displayed in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| <br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| <br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89334PRE2019 3 Group152020-03-29T10:20:39Z<p>S165296: /* A GUI for QET matching */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 has The Effenaar developed into one of the biggest music venues of The Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focusses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are <span style="background:yellow">chaos, energy and tempo</span> <b><span style="background:yellow">ADD HERE MORE FEATURES DURING THE PROJECT</span></b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. Because the algorithm is of value to the project as a whole we worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| <br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| <br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89331PRE2019 3 Group152020-03-29T10:17:52Z<p>S165296: /* Excitement matching with QET */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 has The Effenaar developed into one of the biggest music venues of The Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focusses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are <span style="background:yellow">chaos, energy and tempo</span> <b><span style="background:yellow">ADD HERE MORE FEATURES DURING THE PROJECT</span></b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. However, because the algorithm is of value to the project as a whole we still worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| <br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| <br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89330PRE2019 3 Group152020-03-29T10:17:35Z<p>S165296: /* Excitement matching with QET */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 has The Effenaar developed into one of the biggest music venues of The Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focusses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are <span style="background:yellow">chaos, energy and tempo</span> <b><span style="background:yellow">ADD HERE MORE FEATURES DURING THE PROJECT</span></b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. However, because the algorithm is of value to the project as a whole we still worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
[[File:GUI_QET.jpg|600px|A graphical interface for QET matching]]<br />
<br />
If the user follows through the numbered steps it can totally control the desired QET of his or her music set. In the first step the user defines the duration of the music set in minutes. In the next step the user defines the number of time points he or she wants to control - the depth of control so to speak. In the third step the user makes use of a slider to set the desired excitement values of all the time points. If the user makes any mistakes in this, it is possible to go back to previous points. In the last step the user defines the length that he or she wants to hear all songs and then presses "Generate matching values". When this button is pushed the user gets to see the graph with the interpolation result, polynomial fit and bar graph of the tracks - as described in the prior section. Also, this bar graph of the tracks and their excitement values is transformed into data (an array) and outputted to a ".txt" file so the rest of the system could work with it.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| <br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| <br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=File:GUI_QET.jpg&diff=89328File:GUI QET.jpg2020-03-29T10:07:25Z<p>S165296: </p>
<hr />
<div></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=89327PRE2019 3 Group152020-03-29T10:02:47Z<p>S165296: /* Excitement matching with QET */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===<span style="background:yellow">Primary user needs</span>===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===<span style="background:yellow">Secondary user needs</span>===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 has The Effenaar developed into one of the biggest music venues of The Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focusses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However retrieving information this way could still be valuable. It can be measured on real people how their body reacts to certain music tracks.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on <span style="background:yellow">feedforward</span> based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objectives, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways to obtain feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audience it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are <span style="background:yellow">chaos, energy and tempo</span> <b><span style="background:yellow">ADD HERE MORE FEATURES DURING THE PROJECT</span></b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), multiple ways of interacting are described, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. A simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, <span style="background:yellow">the system can pick from the audience favourite tracks that are popular and valued by the audience,</span> which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b><span style="background:yellow">MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</span></b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ at gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, <span style="background:yellow">valence</span> can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better at gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm <--==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
<!-- invisible text [[File:code_interpolation.jpg|MATLAB code]] --><br />
<nowiki>%Input the key data points (degree of polynomial should be < data points)<br />
QET_matrix = readmatrix('sample_input.txt'); %First row is time, Second row is excitement<br />
x_keydata = QET_matrix(1,:); %The time points from the user input file<br />
y_keydata = QET_matrix(2,:); %The corresponding excitement ratings<br />
p_keydata = polyfit(x_keydata,y_keydata,3); %Get polynomial coefficients for the key data points<br />
f_keydata = polyval(p_keydata,x_keydata); %Generate a fucntion from coefficients<br />
figure;<br />
plot(x_keydata,y_keydata,'o',x_keydata,f_keydata,'-'); %Not a desirable outcome -> use interpolation<br />
title("Input of key data points and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
<br />
%Interpolation for the key data points<br />
samples = max(x_keydata); %#samples to interpolate<br />
x_interpolation = 1:samples;<br />
for i = 1:samples<br />
y_interpolation(i) = interp1(x_keydata,y_keydata,i); %Linear interpolation between input points<br />
end<br />
<br />
%Get a formula for the graph<br />
n = 7; %Degree of the polynomial<br />
p_polynomial = polyfit(x_interpolation,y_interpolation,n); %Get polynomial coefficients for the interpolated data<br />
f_polynomial = polyval(p_polynomial,x_interpolation); %Make a function from these coefficients<br />
<br />
%Create discrete bars (the songs)<br />
songlength = 2; %In minutes<br />
for i = 1:songlength:length(f_polynomial)<br />
track_histogram(i) = f_polynomial(i); %Generate the discrete tracks<br />
end<br />
figure;<br />
hold on;<br />
plot(x_interpolation,y_interpolation,'o',x_interpolation,f_polynomial,'-'); %Plot graphs<br />
alpha(bar(track_histogram,songlength),0.1); %Plot bars<br />
ylim([0.6 1.05]);<br />
title("Interpolation result and polynomial fit");<br />
xlabel("Time [m]");<br />
ylabel("Excitement rating [0,1]");<br />
legend("Interpolated values", "Polynomial curve", "Tracks on the set");<br />
<br />
%Output the formula coefficients<br />
fprintf("Fitted formula coefficients (a1*x^n + a2*x^(n-1) + ...): \n");<br />
for i = 1:n+1<br />
fprintf("%d \n",p_polynomial(i)); %Prints the coefficients<br />
end<br />
<br />
%Output the tracks' excitement rating to a text-file<br />
fileID = fopen('sample_output.txt','w');<br />
fprintf(fileID,"%1.3f\r\n",track_histogram);<br />
fclose(fileID);</nowiki><br />
<br />
==A GUI for QET matching==<br />
Given the time budget it was unfeasible to integrate the QET matching algorithm in the Java-based website. Therefore, we decided to let it function as a standalone application that comes as an extra to our system. However, because the algorithm is of value to the project as a whole we still worked further on it and created an executable GUI for it. The working principles are exactly the same as described in the prior sections, but now it is easier for the user to input his or her desired QET via a visual tool. A screenshot of the tool is provided below.<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
| <br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
| Started working on interview with [[#The Effenaar|the Effenaar]]: <i>1 hour</i><br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
| <br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
| Looked into Webflow. <i>2 hours</i><br />
| <br />
| Worked out interview with [[#The Effenaar|the Effenaar]]: <i>2 hours</i><br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| Attended group meeting <i>1.5 hours</i><br />
| <br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| Improved frontend code, took a Vue,js class <i>5 hours</i><br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| Searched for frontend CSS design & Improved code <i>4 hours</i><br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 23-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 24-03<br />
|<br />
| <br />
| <br />
| <br />
| <br />
|<br />
|-<br />
! scope="row"| Wednesday 25-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 26-03<br />
| We had a meeting in which we discussed the feedback from the tutor session.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i>, rearranging the wiki and improving the display of code on the wiki: <i>45 minutes</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 27-03<br />
| <br />
| <br />
| <br />
| Scanning through the wiki on unsubstantiated choices and other nitpicking: <i>2 hours</i><br />
| <br />
| <br />
|-<br />
! scope="row"| Saturday 28-03<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 29-03<br />
| <br />
| <br />
| <br />
|<br />
| <br />
| <br />
|-<br />
|}<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Retrieved from https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=87601PRE2019 3 Group152020-03-23T09:38:39Z<p>S165296: /* Weekly Log */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===Primary user needs===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===Secondary user needs===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
Since 1971 has The Effenaar developed into one of the biggest music venues of The Netherlands. Having one main hall made for 1300 people and a second hall with a capacity of 400 people. Besides their music halls and restaurant, a part of their group is working on their so called Effenaar Smart Venue. Their goal is to improve the music experience of their guests, and the artists, during their shows. In order to do this, the Smart Venue focusses on existing parts of technology. They ask themselves: what is available? And, what fits in certain shows?<br />
<br />
For example, at a concert of Ruben Hein, they had 12 people wearing sensors measuring brainwaves, heartrates and sweat intensity on their fingertips. As seen in this [https://www.youtube.com/watch?v=_7us-kS0d3E video]. Afterwards, they were able to see which music tracks triggered their enthusiasm and which tracks didn’t. The difference between our project and theirs is that we want to have the user feedback at the same time of the music playback. However this information could still be valuable, as it can be measured on real people which tracks are appreciated. By using another source of information than the Spotify API [LINK], the data can become more reliable.<br />
<br />
Besides only focusing on music, they had several other projects. During one show they had 4 control panels set up in the hall. With these panels the guests could change the visuals of the show. Moreover they had a project where they would make a 3D scan of guests to be used by the artist as visuals or holograms. In another experiment, they introduced an app which would raise the awareness of the importance of wearing earplugs. They used team spirit as a tool to drive this behavior, since they see sense of community as a strong value. They had a project running on feedforward based on a user database, but this was shut down due to it not being financially profitable.<br />
<br />
The Effenaar can still benefit from an improvement of their crowd control. For example, their main hall has a bar right next to the entrance, which sometimes causes blockades. Here, technology could play a role. The crowd could be directed by the use of lighting or what songs are being played.<br />
<br />
Our robot could help the Effenaar in their needs when focusing on music being played before and after the shows. In their second hall, the lights are already automatically adapted on the music that is being played. However the music before and after shows is controlled by an employer of the Effenaar. Most of the time this results in just playing an ordinary Spotify playlist. It would be more convenient if this could be automated and have a music set which fits to the show. They don’t see much potential in replacing the artist of a show by a robot DJ. It would rather result in a collaboration between a DJ and the technology. As said, they emphasize on the social aspect of a night out, to create a memorable moment.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objective, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways of feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
|<br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
|<br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 16-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 17-03<br />
|<br />
| <br />
| Worked further on [[#A script for QET matching algorithm|QET]] track ordering: <i>2 hours</i><br/>Looked into Java MATLAB connection: <i>1 hour</i><br />
| <br />
| Worked on UI <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Wednesday 18-03<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 19-03<br />
| <br />
| <br />
| Worked on an interface for the [[#A script for QET matching algorithm|QET]], now works with .txt files: <i>2 hours</i><br/>Learned gitHub: <i>1 hour</i><br />
| <br />
| Worked on UI <i>5 hours</i><br />
| <br />
|-<br />
! scope="row"| Friday 20-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, and worked on setting up GitHub.<br />
<br/><i> 1.5 hours</i><br />
| <br />
| <br />
|<br />
| Attended the group meeting: <i>1.5 hours</i>, Looked at/trying to understand Mats's improvements on UI <i>30 minutes</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 21-03<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 22-03<br />
| <br />
| <br />
| <br />
|<br />
| Added new inputs and a value display to the [[#User interface | UI]] <i>2 hours</i><br />
| Worked on the user interface using the code Yvonne made. I changed the HTML slightly and added CSS in Adobe Dreamweaver. <i> 2 hours </i><br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audiene it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy and tempo <b>ADD HERE MORE FEATURES DURING THE PROJECT</b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), describes multiple ways of interacting, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. This second solution is a simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
<br/><br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, the system can pick from the audience favourite tracks that are popular and valued by the audience, which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b>MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ in gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, valence can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better in gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
On the user interface (UI), the user can define the input values for different parameters that the pre-filter will use to filter the data-set. <br />
<br />
[[File:UI.PNG|none|500px|First version UI]]<br />
''Figure 1: First UI containing sliders and the input values''<br />
<br />
<br><br />
<br />
On the UI, the input values can be defined by using sliders. The value that the slider is currently on is also displayed on the UI. Then, by clicking on the submit button, the input values are confirmed and put in a JavaScript Object, or Json file. The UI displayed in Fig. 1 is a very primitive version without any styling, which can be done in CSS. In Fig. 2 it can be seen that the correct values were put into the Json file.<br />
<br />
<br/><br />
<br />
[[File:Json.PNG|none|250px|Json file containing parameters]]<br />
''Figure 2: The Json file containing the correct values for the parameters''<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
[[File:code_interpolation.jpg|MATLAB code]]<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Geraadpleegd van https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=86771PRE2019 3 Group152020-03-19T10:58:36Z<p>S165296: /* Excitement matching with QET */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===Primary user needs===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===Secondary user needs===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objective, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways of feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
|<br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
|<br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audiene it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy and tempo <b>ADD HERE MORE FEATURES DURING THE PROJECT</b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), describes multiple ways of interacting, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. This second solution is a simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
<br/><br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, the system can pick from the audience favourite tracks that are popular and valued by the audience, which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b>MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ in gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, valence can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better in gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This input from the user is read from a ".txt" file. This file is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The output of the system is a ".txt" file that contains the excitement ratings of the tracks in the generated set from the user input. The MATLAB script that does all these calculations is provided below.<br />
<br />
[[File:code_interpolation.jpg|MATLAB code]]<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Geraadpleegd van https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=File:Code_interpolation.jpg&diff=86768File:Code interpolation.jpg2020-03-19T10:55:00Z<p>S165296: uploaded a new version of "File:Code interpolation.jpg"</p>
<hr />
<div></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=86646PRE2019 3 Group152020-03-17T11:48:53Z<p>S165296: /* A script for QET matching algorithm */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===Primary user needs===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===Secondary user needs===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objective, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways of feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
|<br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
|<br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audiene it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy and tempo <b>ADD HERE MORE FEATURES DURING THE PROJECT</b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), describes multiple ways of interacting, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. This second solution is a simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
<br/><br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, the system can pick from the audience favourite tracks that are popular and valued by the audience, which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b>MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ in gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, valence can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better in gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. In this example case a duration of 2 minutes is chosen for every song such that a lot of music can be played while there is time enough for every song to play its core part. The MATLAB script that does all these calculations is provided below.<br />
<br />
[[File:code_interpolation.jpg|MATLAB code]]<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Geraadpleegd van https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=86645PRE2019 3 Group152020-03-17T11:43:26Z<p>S165296: /* System overview */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===Primary user needs===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===Secondary user needs===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objective, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways of feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
|<br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
|<br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audiene it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy and tempo <b>ADD HERE MORE FEATURES DURING THE PROJECT</b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), describes multiple ways of interacting, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. This second solution is a simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
<br/><br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, the system can pick from the audience favourite tracks that are popular and valued by the audience, which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b>MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ in gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, valence can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better in gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. The corresponding MATLAB script is provided below.<br />
<br />
[[File:code_interpolation.jpg|MATLAB code]]<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The database is freely available to the user. The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is also freely available to the user. The playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Geraadpleegd van https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=File:System_overview_graphic.JPG&diff=86644File:System overview graphic.JPG2020-03-17T11:41:14Z<p>S165296: uploaded a new version of "File:System overview graphic.JPG"</p>
<hr />
<div></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=File:Code_interpolation.jpg&diff=86643File:Code interpolation.jpg2020-03-17T11:25:55Z<p>S165296: uploaded a new version of "File:Code interpolation.jpg"</p>
<hr />
<div></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=86642PRE2019 3 Group152020-03-17T11:22:19Z<p>S165296: /* A script for QET matching algorithm */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===Primary user needs===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===Secondary user needs===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objective, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways of feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
|<br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
|<br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audiene it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy and tempo <b>ADD HERE MORE FEATURES DURING THE PROJECT</b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), describes multiple ways of interacting, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. This second solution is a simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
<br/><br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, the system can pick from the audience favourite tracks that are popular and valued by the audience, which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b>MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ in gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, valence can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better in gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. These tracks are depicted as the orange bars in the graph. The corresponding MATLAB script is provided below.<br />
<br />
[[File:code_interpolation.jpg|MATLAB code]]<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Geraadpleegd van https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=86641PRE2019 3 Group152020-03-17T11:19:11Z<p>S165296: /* A script for QET matching algorithm */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===Primary user needs===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===Secondary user needs===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
==The Effenaar==<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objective, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways of feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on the Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with the Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| Attended the group meeting: <i>1 hour</i><br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| Ideated extra backend implementations: <i>1 hour</i> <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| Worked on a more general grouping of the different genres and worked on the visualization of a part of the back end <i>45 minutes</i><br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
|<br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
|<br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audiene it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy and tempo <b>ADD HERE MORE FEATURES DURING THE PROJECT</b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), describes multiple ways of interacting, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. This second solution is a simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
<br/><br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, the system can pick from the audience favourite tracks that are popular and valued by the audience, which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b>MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ in gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, valence can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better in gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by [[#The Effenaar|the Effenaar]]. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET_including_bars.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. The MATLAB script is provided below.<br />
<br />
[[File:code_interpolation.jpg|MATLAB code]]<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#The Effenaar|the Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Geraadpleegd van https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=File:Interpolation_QET_including_bars.jpg&diff=86640File:Interpolation QET including bars.jpg2020-03-17T11:18:34Z<p>S165296: </p>
<hr />
<div></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=File:Interpolation_QET.jpg&diff=86639File:Interpolation QET.jpg2020-03-17T11:17:18Z<p>S165296: uploaded a new version of "File:Interpolation QET.jpg":&#32;Reverted to version as of 11:14, 17 March 2020</p>
<hr />
<div></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=File:Interpolation_QET.jpg&diff=86638File:Interpolation QET.jpg2020-03-17T11:16:24Z<p>S165296: uploaded a new version of "File:Interpolation QET.jpg":&#32;Reverted to version as of 11:14, 17 March 2020</p>
<hr />
<div></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=File:Interpolation_QET.jpg&diff=86637File:Interpolation QET.jpg2020-03-17T11:15:44Z<p>S165296: uploaded a new version of "File:Interpolation QET.jpg"</p>
<hr />
<div></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=File:Interpolation_QET.jpg&diff=86636File:Interpolation QET.jpg2020-03-17T11:14:35Z<p>S165296: uploaded a new version of "File:Interpolation QET.jpg"</p>
<hr />
<div></div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=86403PRE2019 3 Group152020-03-15T20:01:47Z<p>S165296: /* Weekly Log */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===Primary user needs===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===Secondary user needs===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objective, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways of feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on de Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
|<br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
|<br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a [[#Excitement matching with QET|QET matching algorithm]]: <i>2.5 hours</i><br />
|<br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audiene it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy and tempo <b>ADD HERE MORE FEATURES DURING THE PROJECT</b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), describes multiple ways of interacting, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. This second solution is a simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
<br/><br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, the system can pick from the audience favourite tracks that are popular and valued by the audience, which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b>MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ in gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, valence can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better in gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by de Effenaar <b>REFERENTIE NAAR UITWERKING GESPREK MET EFFENAAR</b>. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. The MATLAB script is provided below.<br />
<br />
[[File:code_interpolation.jpg|MATLAB code]]<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#LINK EFFENAAR|(link naar uitwerking gesprek) De Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Geraadpleegd van https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=86402PRE2019 3 Group152020-03-15T20:00:43Z<p>S165296: /* Weekly Log */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===Primary user needs===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===Secondary user needs===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objective, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways of feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on de Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
|<br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| <br />
| Creating a graphical [[#System overview|system overview]]: <i>2 hours</i><br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the [[#System overview|system overview]] on the Wiki: <i>2 hours</i><br />
|<br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a QET matching algorithm: <i>2.5 hours</i><br />
|<br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audiene it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy and tempo <b>ADD HERE MORE FEATURES DURING THE PROJECT</b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), describes multiple ways of interacting, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. This second solution is a simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
<br/><br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, the system can pick from the audience favourite tracks that are popular and valued by the audience, which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b>MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ in gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, valence can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better in gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by de Effenaar <b>REFERENTIE NAAR UITWERKING GESPREK MET EFFENAAR</b>. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. The MATLAB script is provided below.<br />
<br />
[[File:code_interpolation.jpg|MATLAB code]]<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#LINK EFFENAAR|(link naar uitwerking gesprek) De Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Geraadpleegd van https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=86401PRE2019 3 Group152020-03-15T19:59:19Z<p>S165296: /* System overview (NEEDS EDITING) */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===Primary user needs===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===Secondary user needs===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objective, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways of feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on de Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
|<br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| <br />
| Creating a graphical system overview: <i>2 hours</i><br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the system overview on the Wiki: <i>2 hours</i><br />
|<br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a QET matching algorithm: <i>2.5 hours</i><br />
|<br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audiene it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy and tempo <b>ADD HERE MORE FEATURES DURING THE PROJECT</b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), describes multiple ways of interacting, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. This second solution is a simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
<br/><br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, the system can pick from the audience favourite tracks that are popular and valued by the audience, which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b>MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ in gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, valence can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better in gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by de Effenaar <b>REFERENTIE NAAR UITWERKING GESPREK MET EFFENAAR</b>. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. The MATLAB script is provided below.<br />
<br />
[[File:code_interpolation.jpg|MATLAB code]]<br />
<br />
=System overview=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#LINK EFFENAAR|(link naar uitwerking gesprek) De Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Geraadpleegd van https://core.ac.uk/reader/77040676</div>S165296https://cstwiki.wtb.tue.nl/index.php?title=PRE2019_3_Group15&diff=86400PRE2019 3 Group152020-03-15T19:59:06Z<p>S165296: /* System overview (NEEDS EDITING) */</p>
<hr />
<div>== Group Members ==<br />
<br />
{| class="wikitable" style="border-style: solid; border-width: 1px;" cellpadding="3"<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Study<br />
!style="text-align:left;"| Student ID<br />
|-<br />
| Mats Erdkamp || Industrial Design|| 1342665<br />
|- <br />
| Sjoerd Leemrijse|| Psychology & Technology || 1009082<br />
|-<br />
| Daan Versteeg || Electrical Engineering || 1325213<br />
|-<br />
| Yvonne Vullers|| Electrical Engineering|| 1304577<br />
|-<br />
| Teun Wittenbols|| Industrial Design|| 1300148<br />
|}<br />
<br />
=Problem Statement and Objectives=<br />
DJ-ing is a relatively new profession. It has only been around for less than a century but has become more and more widespread in the last few decades. This activity has for the most part been executed by human beings. Current technology in the music industry has become better and better at generating playlists, or 'recommended songs' as, for example, Spotify does. Can we integrate a form of this technology into the world of DJs and create a 'robot DJ'? A robot DJ would autonomously create playlists and mix songs, based on algorithms and real-life feedback in order to entertain an audience.<br />
<br />
How to develop an autonomous system/robot DJ which enables the user to easily use it as a substitute for a human DJ.<br />
<br />
=Users=<br />
The identification of the primary and secondary users and their needs is based on extensive literature research on the interaction between the DJ and the audience in a party setting. The reader is referred to the [[#References|references]] section for the full articles. This section is written based on [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]] and [[#References|(Berkers & Michael, 2017)]].<br />
==Primary users==<br />
*Dance industry: this is the overarching organization that will possess most of the robots.<br />
*Organizer of a music event: this is the user that will rent or buy the robot to play at their event.<br />
*Owner of a discotheque or club: the robot can be an artificial alternative for hiring a DJ every night.<br />
<br />
===Primary user needs===<br />
*The DJ-robot is more valued than a human performer.<br />
**The DJ-robot system provides something extraordinary and special.<br />
**The system is better than a human DJ in gathering information regarding audience appreciation.<br />
*The system's user interface is easy to understand, no experts needed.<br />
<br />
==Secondary users==<br />
*Attenders of a music event: these people enjoy the music and lighting show that the robot makes.<br />
*Human DJ's: likely to "cooperate" with a DJ-robot to make their show more attractive.<br />
<br />
===Secondary user needs===<br />
*The music set played is structured and progressive.<br />
**Track selection should fit the audience background.<br />
***The system selects appropriate tracks regarding genre.<br />
***The musical presentation should reflect the audience energy level.<br />
**There is a balance between playing familiar tracks and providing rare, new music.<br />
***The system selects popular tracks that are valued by the audience.<br />
***Similar tracks to what the audience is into are played.<br />
**The dancers are taken on a cohesive and dynamical music journey.<br />
<br />
<br />
*The audience desires control over the music being played, to a certain extent.<br />
**The audience wants to hear their favourite music.<br />
**The audience doesn't want a predictable set of music.<br />
**The DJ-robot takes the audience reaction into account in track selection.<br />
<br />
=Approach, Milestones, and Deliverables=<br />
==Approach==<br />
The goal of the project is to create a robot that functions as a DJ and provides entertainment to a crowd. In order to reach the goal, first a literature study will be executed to find out the current state of the art regarding the problem. After enough information has been collected, an objective will be defined.<br />
<br />
Then, the USE and technical aspects of the problem will be researched.<br />
The technical aspect-research will be implemented in a design for the robot.<br />
Based on this design a prototype will be built and programmed that is able to meet the requirements of the goal.<br />
<br />
==Milestones==<br />
In order to complete the project and meet the objective, milestones have been determined. These milestones include:<br />
* A clear problem and goal have been determined<br />
* The literature research is finished, this includes research about<br />
** Users (attenders of music events, DJs, club owners)<br />
** The state of the art in the music (AI) industry<br />
* The research on how to create the DJ-software is finished<br />
** Ways of feedback from a crowd<br />
** Spotify API<br />
* The DJ-software is created<br />
** Depending on what method of feedback is chosen, a sensor is also built<br />
* A test is executed in which the environment the software will be used in is simulated<br />
* The wiki is finished and contains all information about the project<br />
<br />
<br />
Other milestones, which probably are not attainable in the scope of 8 weeks, are<br />
* A test in a larger environment is executed (bar, festival)<br />
* A full scale robot is constructed to improve the crowd’s experience<br />
* A light show is added in order to improve the crowd’s experience<br />
<br />
==Deliverables==<br />
The deliverables for this project are:<br />
* The DJ-software, which is able to use feed-forward and incorporate user feedback in order to create the most entertaining DJ-set<br />
* Depending on what type of user feedback is chosen, a prototype/sensor also needs to be delivered<br />
* The wiki-page containing all information on the project<br />
* The final presentation in week 8<br />
<br />
=Who's Doing What?=<br />
<br />
==Personal Goals== <br />
The following section describes the main roles of the teammates within the design process. Each team member has chosen an objective that fits their personal development goals.<br />
<br />
{| class="wikitable" style=<br />
!style="text-align:left;"| Name<br />
!style="text-align:left"| Personal Goal<br />
|-<br />
| Mats Erdkamp || Play a role in the development of the artificial intelligence systems.<br />
|- <br />
| Sjoerd Leemrijse || Gain knowledge in recommender systems and pattern recognition algorithms in music.<br />
|-<br />
| Daan Versteeg || <br />
|-<br />
| Yvonne Vullers || Play a role in creating the prototype/artificial intelligence<br />
|-<br />
| Teun Wittenbols || Combine all separate parts into one good concept, with a focus on user interaction. <br />
|}<br />
<br />
==Weekly Log==<br />
Based on the approach and the milestones, a planning has been made. This planning is not definite and will be updated regularly, however it will be a guideline for the coming weeks.<br />
<br />
<br/><br />
<b>Week 2</b><br />
Goal: Do literature research, define problem, make a plan, define users, start research into design and prototype<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 10-02<br />
| We formed a group and discussed the first possibilities within the project, chose a general theme and started doing research. We attended the tutor session.<br />
<br/><i>30 minutes + 30 minutes</i><br />
| Work on SotA and evaluate design options<br />
| Attended meeting with tutors: <i>30 minutes</i><br/>Elaborated the notes of the meeting: <i>30 minutes</i><br/>Reading and summarizing scientific literature on the interaction between DJ and crowd: <i>3 hours</i><br />
| Attended meeting with tutors: <i>30 minutes</i><br />
| Attended tutor meeting <br />
<br/><i>0.5 hours</i><br />
| Started doing literature research and summarized [[#Summary of Related Research|Pasick (2015) & Johnson]]. Formed a group and attended meeting.<br />
<br/><i>2 hours + 30 minutes + 30 minutes</i><br />
|-<br />
! scope="row"| Tuesday 11-02<br />
| <br />
|<br />
| Searching scientific literature on user requirements of the public and the DJ at a party [[#References|(Gates, Subramanian & Gutwin, 2006)]], [[#References|(Gates & Subramanian, 2006)]]: <i>1 hour</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 12-02<br />
| <br />
|<br />
| Summarizing scientific literature on user requirements: <i>3 hours</i><br />
|<br />
| Started looking for papers about user interaction and user feedback<br />
<br/><i> 2 hours </i><br />
|<br />
|-<br />
! scope="row"| Thursday 13-02<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week: <i>1.5 hours</i><br />
|<br />
| Meeting with group members, discussing who will be doing what the coming week. Emailed Effenaar.<br />
<br/><i> 1.5 hours</i><br />
| Did some more research on audience interaction and summerized it. [[#Summary of Related Research|(Hödl, Fitzpatrick, Kayali & Holland, 2017)(Zhang, Wu, & Barthet, ter perse)]] I, also updated the wiki, made the planning more clear and divided the references of the SotA. Attended the meeting.<br />
<br/><i> 1.5 hours + 45 minutes + 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 14-02<br />
| <br />
| Developed access to spotify API ''4 hours.''<br />
|<br />
|<br />
|<br />
| Researched different forms of audience interaction and added it to the SotA. [[#Receiving feedback from the audience via technology]] <br />
<br/><i>1.5 hours</i><br />
|-<br />
! scope="row"| Saturday 15-02<br />
| <br />
| Created data set from spotify API: ''8 hours''<br />
| Writing the [[#Users|"Users"]] section based on my prior literature research: <i>2 hours</i><br/>Updating the [[#State of the Art|section]] on state of the art based on my prior literature research: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 16-02<br />
| <br />
|<br />
| Updating the lay-out of the wiki: <i>1 hour</i><br />
|<br />
| Updating the [[#Milestones|milestones]] and [[#Deliverables|deliverables]]. Continued looking for papers on incorporating user feedback and user feedback for music events. Summarized papers [[#Summary of Related Research|(Barkhuus & Jorgensen, 2008)]] & [[#Summary of Related Research|(Atherton, Becker, McLean, Merkin & Rhoades, 2008)]]<br />
<br/><i> 2.5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 3</b><br />
Goal: Continue research, start on design<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 17-02<br />
| Group meeting and general discussion: <i>45 minutes</i><br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| Attended tutor meeting: <i>30 minutes</i>. Discussion with group members: <i>15 minutes</i>. <br />
| <br />
| Attended tutor meeting: <br />
<br/><i> 45 min</i><br />
|-<br />
! scope="row"| Tuesday 18-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 19-02<br />
| <br />
|<br />
| Worked on a first concept model of our designed system, [[#A first model|A first model]]. <i>4 hours</i><br />
|<br />
| Elaborated the notes of the meeting: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 20-02<br />
| Group meeting, discussing plans and everyone's contributions: <i>1 hour</i><br />
| Group meeting, discussed plans <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i>. Worked on how the user needs lead to a [[#A first model|first model]]: <i>2 hours</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <i>1 hour</i><br />
| Attended group meeting: <br />
<br/><i>1 hour</i><br />
|-<br />
! scope="row"| Friday 21-02<br />
| <br />
| <br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 22-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 23-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Carnaval break</b><br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 24-02<br />
| <br />
| <br />
| Worked on a schematic to show how the user needs relate to our design: <i>1.5 hours</i><br />
| <br />
| <br />
| <br />
|-<br />
! scope="row"| Tuesday 25-02<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 26-02<br />
| <br />
|<br />
| <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 27-02<br />
| <br />
|<br />
| Worked on a model to predict music parameters using the Spotify dataset in a multiple regression model: <i>2 hours</i><br />
|<br />
|<br />
| Get to know the basics of node.js <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 28-02<br />
| <br />
| <br />
|<br />
|<br />
| Worked on getting to know javascript, node.js, and visual studio: <i>6 hours</i>.<br />
| <br />
|-<br />
! scope="row"| Saturday 28-02<br />
| <br />
| <br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 1-03<br />
| <br />
| Started integration of data set in Tempo curve algorhithm <i>4 hours</i>.<br />
| Did research on de Effenaar: <i>30 minute</i> <br/> Elaborating on the algorithm in [[#A first model|the first model]]: <i>2 hours</i><br />
|<br />
| Continued on getting to know javascript and node.js. Also started on getting to know express JS: <i>4 hours</i>.<br />
|<br />
|-<br />
|}<br />
<br/><br />
<b>Week 4</b><br />
Goal: Finish first design, start working on software.<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 02-03<br />
| We attended the tutor session and went to the Effenaar in order to get some general information about the possibilities, and or user needs.<br />
<br/><i>30 minutes + 1 hour </i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br/>Worked out the notes on the conversation with Effenaar: <i>30 minutes</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br/>Had a conversation with de Effenaar: <i>1 hour</i><br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 03-03<br />
| <br />
|<br />
| Worked on explaining the feedforward and feedback parameters more clearly in [[#A first model|the concept model]]: <i>2 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 04-03<br />
| <br />
|<br />
|Did research on the Spotify audio features, tried to come up with exact definitions: <i>2 hours</i> <br />
|<br />
| <br />
|<br />
|-<br />
! scope="row"| Thursday 05-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, discussed the research and formed a more detailed and specific plan for the project.<br />
<br/><i> 1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the group meeting: <i>1.5 hours</i><br />
| Attended the meeting <br />
<br/><i> 1.5 hours</i><br />
|-<br />
! scope="row"| Friday 06-03<br />
| <br />
| Cleaned up data set generation code <i> 2 hours</i><br />
|<br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 07-03<br />
| <br />
| Added new tags to data set & included pre-filtering backend. <i> 7 hours</i><br />
| Worked on a multiple regression model for feedforward: <i>4 hours</i><br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Sunday 08-03<br />
| <br />
| Finalized pre-filtering backend + made API calls more reliable <i> 4 hours</i><br />
| Processed the results of the multiple regression analysis and added them to the [[#Excitement prediction with multiple regression|wiki]]: <i>3 hours</i><br />
|<br />
| Worked more on node.js, expressJS and the UI: <i>6 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br />
<b>Week 5</b><br />
Goal: Work on software<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
|<br />
! scope="col"| Group<br />
! scope="col"| Mats Erdkamp<br />
! scope="col"| Sjoerd Leemrijse<br />
! scope="col"| Daan Versteeg<br />
! scope="col"| Yvonne Vullers<br />
! scope="col"| Teun Wittenbols<br />
|-<br />
! scope="row"| Monday 09-03<br />
| We attended the tutor session <br/><i>30 minutes</i><br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i> <br />
| <br />
| Attended the tutor meeting: <i>30 minutes</i><br />
| <br />
|-<br />
! scope="row"| Tuesday 10-03<br />
| <br />
|<br />
| <br />
|<br />
|<br />
|<br />
|-<br />
! scope="row"| Wednesday 11-03<br />
| <br />
|<br />
| Try to specify/quantify "excitement" and search for data on it: <i>2 hours</i><br />
|<br />
| Elaborated meeting notes: <i>30 minutes</i><br />
|<br />
|-<br />
! scope="row"| Thursday 12-03<br />
| We had a meeting in which we discussed the feedback from the tutor session, formed a more detailed and specific plan for the project.<br />
<br/><i> 1 hour</i><br />
|<br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
| Attended the group meeting: <i>1 hour</i><br />
| <br />
|-<br />
! scope="row"| Friday 13-03<br />
| <br />
| <br />
| Creating a graphical system overview: <i>2 hours</i><br />
|<br />
|<br />
| <br />
|-<br />
! scope="row"| Saturday 14-03<br />
| <br />
| <br />
| Describing the system overview on the Wiki: <i>2 hours</i><br />
|<br />
| Worked on getting to know Bootstrap JS for creating the UI: <i>2 hours</i><br />
|<br />
|-<br />
! scope="row"| Sunday 15-03<br />
| <br />
| <br />
| Worked on a QET matching algorithm: <i>2.5 hours</i><br />
|<br />
| Worked on learning parts that are useful for creating the UI & a little bit on the UI itself: <i>5 hours</i><br />
|<br />
|-<br />
|}<br />
<br/><br />
<br/><br />
<b>Week 6</b><br />
Goal: Finish software , do testing<br />
<br/><br />
<b>Week 7</b><br />
Goal: Finish up the last bits of the software<br />
<br/><br />
<b>Week 8</b><br />
Goal: Finish wiki, presentation<br />
<br />
=State of the Art=<br />
The dance industry is a booming business that is very receptive of technological innovation. A lot of research has already been conducted on the interaction between DJ's and the audience and also in automating certain cognitively demanding tasks of the DJ. Therefore, it is necessary to give a clear description of the current technologies available in this domain. In this section the state of the art on the topics of interest when designing a DJ-robot are described by means of recent literature.<br />
<br />
==Defining and influencing characteristics of the music== <br />
When the system receives feedback from the audiene it is necessary that it is also able to do something useful with that feedback and convert it into changes in the provided musical arrangement. The most important aspects of this arrangement are chaos, energy and tempo <b>ADD HERE MORE FEATURES DURING THE PROJECT</b>.<br />
<br />
Chaos is the opposite of order. Dance tracks with order can be assumed as having a repetitive rhythmic structure, contain only a few low-pitched vocals and display a pattern in arpeggiated lines and melodies. Chaos can be created by undermining these rules. Examples are playing random notes, changing the rhythmic elements at random time-points, or increasing the number of voices present and altering their pitch. Such procedures create tension in the audience. This is necessary because without differential tension, there is no sense of progression [[#References|(Feldmeier, 2003)]].<br />
<br />
The factors energy and tempo are inherently linked to each other. When the tempo increases, so does the perceived energy level of the track. In general, the music's energy level can be intensified by introducing high-pitched, complex vocals and a strong occurence of the beat. Related to that is the activity of the public. An increase in activity of the audience can signal that they are enjoying the current music, or that they desire to move on to the next energy level [[#References|(Feldmeier, 2003)]]. Because in general, people enjoy the procedure of tempo activation in which they dance to music that leads their current pace [[#References|(Feldmeier & Paradiso, 2004)]].<br />
<br />
==Algorithms for track selection==<br />
One of the most important tasks of a DJ-robot - if not the most important - is track selection. In [[#References|(Cliff, 2006)]] a system is described that takes as input a set of selected tracks and a qualitative tempo trajectory (QTT). A QTT is a graph of the desired tempo of the music during the set with time on the x-axis and tempo on the y-axis. Based on these two inputs the presented algorithm basically finds the sequence of tracks that fit the QTT best and makes sure that the tracks construct a cohesive set of music. The following order is taken in this process: track mapping, trajectory specification, sequencing, overlapping, time-stretching and beat-mapping, crossfading.<br />
<br />
In this same article, use is been made of genetic algorithms to determine the fitness of each song to a certain situation. This method presents a sketch of how to encode a multi-track specification of a song as the genes of the individuals in a genetic algorithm [[#References|(Cliff, 2006)]].<br />
<br />
==Receiving feedback from the audience via technology==<br />
Audiences of people attending musical events generally like the idea of being able to influence performance (Hödl et al., 2017). What is important is the way of interaction with the performers, what works well and what do users prefer?<br />
<br />
In the article of Hödl et al. (2017), describes multiple ways of interacting, namely, mobile devices, as can be seen in the article of McAllister et al. (2004), smartphones and other sensory mechanisms, such as the light sticks discussed in the paper of Freeman (2005).<br />
<br />
The system presented by Cliff [[#References|(Cliff, 2006)]] already proposes some technologies that enables the crowd to give feedback to a DJ-robot. They discuss under-floor pressure sensors that can sense how the audience is divided over the party area. They also discuss video surveillance that can read the crowd's activity and valence towards the music. Based on this information the system determines whether the assemblage of the music should be changed or not. In principle, the system tries to stick with the provided QTT, however, it reacts dynamically on the crowd's feedback and may deviate from the QTT if that is what the public wants.<br />
<br />
Another option for crowd feedback discussed by Cliff is a portable technology, more in the spirit of a wristband. One option is a wristband that is more quantitative in nature and transmits location, dancing movements, body temperature, perspiration rates and heart rate to the system. Another option is a much simpler and therefore cheaper solution. This second solution is a simple wristband with two buttons on it; one "positive" button and one "negative" button. In that sense, the public lets the system know whether they like the current music or not, and the system can react upon it.<br />
<br />
==Summary of Related Research==<br />
<br/><br />
This patent describes a system, something like a personal computer or an MP3 player, which incorporates user feedback in it's music selection. The player has access to a database and based on user preferences it chooses music to play. When playing, the user can rate the music. This rating is taken into account when choosing the next song. (Atherton, Becker, McLean, Merkin & Rhoades, 2008)<br />
<br />
<br/><br />
This article describes how rap-battles incorporate user feedback. By using a cheering meter, the magnitude of enjoyment of the audience can be determined. This cheering meter was made by using Java's Sound API. (Barkhuus & Jorgensen, 2008)<br />
<br />
<br/><br />
Describes a system that transcribes drums in a song. Could be used as input for the DJ-robot (light controls for example). (Choi & Cho, 2019)<br />
<br />
<br/><br />
This paper is meant for beginners in the field of deep learning for MIR (Music Information Retrieval). This is a very useful technique in our project to let the robot gain musical knowledge and insight in order to play an enjoyable set of music. (Choi, Fazekas, Cho & Sandler, 2017)<br />
<br />
<br/><br />
This article describes different ways on how to automatically detect a pattern in music with which it can be decided what genre the music is of. By finding the genre of the music that is played, it becomes easier to know whether the music will fit the previously played music.(De Léon & Inesta, 2007)<br />
<br />
<br/><br />
This describes "Glimmer", an audience interaction tool consisting of light sticks which influence live performers. (Freeman, 2005)<br />
<br />
<br/><br />
Describes the creation of a data set to be used by artificial intelligence systems in an effort to learn instrument recognition. (Humphrey, Durand & McFee, 2018)<br />
<br />
<br/><br />
This describes the methods to learn features of music by using deep belief networks. It uses the extraction of low level acoustic features for music information retrieval (MIR). It can then find out e.g. of what genre the the musical piece was. The goal of the article is to find a system that can do this automatically. (Hamel & Eck, 2010)<br />
<br />
<br/><br />
This article communicates the results of a survey among musicians and attenders of musical concerts. The questions were about audience interaction. <i>"... most spectators tend to agree more on influencing elements of sound (e.g. volume) or dramaturgy (e.g. song selection) in a live concert. Most musicians tend to agree on letting the audience participate in (e.g. lights) or dramaturgy as well, but strongly disagree on an influence of sound."</i> (Hödl, Fitzpatrick, Kayali & Holland, 2017)<br />
<br />
<br/> <br />
This article explains the workings of the musical robot Shimon. Shimon is a robot that plays the marimba and chooses what to play based on an analysis of musical input (beat, pitch, etc.). The creating of pieces is not necessarily relevant for our problem, however choosing the next piece of music is of importance. Also, Shimon has a social-interactive component, by which it can play together with humans. (Hoffman & Weinberg, 2010)<br />
<br />
<br/><br />
This article introduces Humdrum, which is software with a variety of applications in music. One can also look at humdrum.org. Humdrum is a set of command-line tools that facilitates musical analysis. It is used often in for example Pyhton or Cpp scripts to generate interesting programs with applications in music. Therefore, this program might be of interest to our project. (Huron, 2002)<br />
<br />
<br/><br />
This article focuses on next-track recommendation. While most systems base this recommendation only on the previously listened songs, this paper takes a multi-dimensional (for example long-term user preferences) approach in order to make a better recommendation for the next track to be played. (Jannach, Kamehkhosh & Lerche, 2017)<br />
<br />
<br/><br />
In this interview with a developer of the robot DJ system POTPAL, some interesting possibilities for a robot system are mentioned. For example, the use of existing top 40 lists, 'beat matching' and 'key matching' techniques, monitoring of the crowd to improve the music choice and to influence people's beverage consumption and more. Also, a humanoid robot is mentioned which would simulate a human DJ. (Johnson, n.a.)<br />
<br />
<br/><br />
In this paper a music scene analysis system is developed that can recognize rhythm, chords and source-separated musical notes from incoming music using a Bayesian probability network. Even though 1995 is not particularly state-of-the-art, these kinds of technology could be used in our robot to work with music. (Kashino, Nakadai, Kinoshita, & Tanaka, 1995)<br />
<br />
<br/><br />
This paper discusses audience interaction by use of hand-held technological devices. (McAllister, Alcorn, Strain & 2004)<br />
<br />
<br/><br />
This article discusses the method by which Spotify generates popular personalized playlists. The method consists of comparing your playlists with other people's playlists as well as creating a 'personal taste profile'. These kinds of things can be used by our robot DJ by, for example, creating a playlist based on what kind of music people listen to the most collectively. It would be interesting to see if connecting peoples Spotify account to the DJ would increase performance. (Pasick, 2015)<br />
<br />
<br/><br />
This paper takes a mathematical approach in recommending new songs to a person, based on similarity with the previously listened and rated songs. These kinds of algorithms are very common in music systems like Spotify and of utter use in a DJ-robot. The DJ-robot has to know which songs fit its current set and it therefore needs these algorithms for track selection. (Pérez-Marcos & Batista, 2017)<br />
<br />
<br/><br />
This paper describes the difficulty of matching two musical pieces because of the complexity of rhythm patterns. Then a procedure is determined for minimizing the error in the matching of the rhythm. This article is not very recent, but it is very relevant to our problem. (Shmulevich, & Povel, 1998)<br />
<br />
<br/><br />
In this article, the author states that the main melody in a piece of music is a significant feature for music style analysis. It proposes an algorithm that can be used to extract the melody from a piece and the post-processing that is needed to extract the music style. (Wen, Chen, Xu, Zhang & Wu, 2019)<br />
<br />
<br/><br />
This research presents a robot that is able to move according to the beat of the music and is also able to predict the beats in real time. The results show that the robot can adjust its steps in time with the beat times as the tempo changes. (Yoshii, Nakadai, Torii, Hasegawa, Tsujino, Komatani, Ogata & Okuno, 2007)<br />
<br />
<br/><br />
This paper describes Open Symphony, a web application that enables audience members to influence musical performances. They can indicate a preference for different elements of the musical composition in order to influence the performers. Users were generally satisfied and interested in this way of enjoying the musical performance and indicated a higher degree of engagement. (Zhang, Wu, & Barthet, ter perse)<br />
<br />
=A first model=<br />
Based on the state of the art and the user needs a first model of our automated DJ system is made. We chose to depict it in a block diagram with separate blocks for the feedforward, the feedback and the algorithm itself.<br />
<br />
[[File:first_model.jpg|none|600px|First model of the automated DJ system]]<br />
<br />
==Feedforward==<br />
The feedforward part of our system is completely based on the user input. The user has a lot of knowledge about the desired DJ set to be played beforehand. This information is fed to the system to control it. Because this feedforward block is based on user input, it has to answer to the user needs. Below, a scheme is presented to show how the user needs relate to the feedforward parameters.<br />
<br />
[[File:feedforward.jpg|none|800px|Feedforward parameters in relation to the user needs]]<br />
<br />
The first part of the feedforward is the desired QTT of the set. What a QTT is, is described in the [[#Algorithms for track selection|state of the art]] section. The ability for the user to input a QTT answers to the primary user need of an easy user interface. Providing a QTT to the system also makes it easy for the system to play a structured and progressive set of music, which is a secondary user need. This is in line with the secondary user need that dancers are taken on a cohesive and dynamical music journey. This structure in the music will also keep the audience engaged. <br />
<br />
To fulfill the desired QTT certain tracks are delivered to the system to pick from as feedforward. These tracks are delivered via an enormous database with all kinds of music in it, such that the system has enough options to pick from in order to form the best set. In that sense, the system can pick from the audience favourite tracks that are popular and valued by the audience, which are some of our secondary user needs. Since people come to certain music events with certain expectations, the tracks to pick from should be appropriate regarding genre. That way, the audience background is taken into account and it keeps them engaged. This is also an opportunity to limit the algorithm's options for track selection, making the system more stable. Because the database the system can choose from is very large and diverse, the user need of wanting to hear rare, new music is answered. Also, this contributes to an unpredictable set of music that is not boring. <br />
<br />
<b>MATS/SJOERD, KAN JIJ MISSCHIEN UITEINDELIJK WAT SCHRIJVEN OVER DE BELANGRIJKSTE SPOTIFY FEATURES DIE WE GAAN GEBRUIKEN ALS FF?</b><br />
<br />
In order to satisfy the audience we have to feedforward background information of the audience members to the system. In that way the system has knowledge about what the current audience is into. What someone is into could include their preferences regarding audience features based on their Spotify profile and their favourite or most hated tracks. This answers to the primary user needs of making the system something extraordinary and more valued than a human DJ because a human DJ can never know this information of every audience member. This also makes that the system is better than a human DJ in gathering information regarding audience appreciation, which is also a primary user need. It also answers to the secondary user needs of keeping balance between familiar and new music (because the system knows which tracks are familiar) and audience members wanting to express themselves - be it by providing information to a computer. It also values the DJ's desire to play similar tracks to what the audience is into and the desire to create a collective experience by means of a music set. This collective is generated because every audience member contributes a part to the feedforward of the system.<br />
<br />
==Feedback==<br />
The feedback of the system consists of sensor output. This is the part where the audience takes more control. The feedback sensor system detects how many persons are present on the dance floor, relative to the rest of the event area. This can be done by means of pressure sensors in the floor. This information cues whether the current music is appreciated or not. Another cue for appreciation of the music can be generated via active feedback of the crowd. For example, valence can be assessed by the public by means of technologies described in the [[#State of the Art|state of the art]] section. Because audience feedback is not commonly used at music events, the primary user need of providing something special and extraordinary is answered. The ability for the crowd to give feedback answers the primary user need that the system should be better in gathering information regarding audience appreciation than a human DJ. This ability also lets the public as a whole have influence on the music, which creates a collective experience. Additionally, it answers the secondary user need of having control over the music being played. In that sense, the track selection procedure takes the audience reaction into account such that it comes up with tracks that are valued by the audience members. This also lets the public control whether the music is predictable or not, which should keep them engaged. <br />
<br />
The other part of sensory feedback is the audience energy level. This energy level may include the activity or movement of the audience members. This can be measured passively by means of a wristband with different options for sensing activity or energy by means of heart rate, accelerometer data, sweat response or other options described in the [[#State of the Art|state of the art]] section. The incorporation of audience energy level feedback answers to the primary user need of gathering information about the audience appreciation in a better way than a human DJ can. It also answers to the secondary user need of making the musical presentation reflect the audience energy level.<br />
<br />
Below, a scheme is presented with all the user needs that call for feedback sensors.<br />
<br />
[[File:feedback_sensors.jpg|none|800px|User needs in relation to feedback sensors]]<br />
<br />
==The algorithm==<br />
Based on the feedforward and feedback of the crowd, the tracks to be played and their sequence are selected as described [[#Excitement matching with QET|here]].<br />
<br />
The next step in the algorithm is overlapping the tracks in the right way. Properly working algorithms that handle this task already exist. For example, the algorithm described in [[#References|(Cliff, 2006)]]. We will describe the working principles of that algorithm in this section. The overlap section is meant to seamlessly go from one track to the other. In the described technology the time set for overlap is proportional to the specified duration of the set and the number of tracks, making it a static time interval. Alterations to the duration of this interval are made when the tempo maps of the overlapping tracks produce a beat breakdown or when the overlap interval leads to an exceedance of the set duration.<br />
<br />
If the system wants to play a next track in a smooth transition and there is any difference between the tempo (BPM) of the current track and the next track, time-stretching and beat-mapping need to be applied. Time-stretching means to slow down or bring up the tempo such that the tempo of the current and next track are nearly identical in order to produce a smooth transition to the next track. Technically speaking time-stretching is a (de)compression of time or changing the playback speed of the samples and applying proper interpolation in order to maintain sound quality. Once the tempos match, the beats of the two songs need to be aligned in order to acquire zero phase difference to avoid beat breakdown.<br />
<br />
The last step is proper cross-fading. Although ramping down the volume of the current track while ramping up the volume of the next track is often sufficient for a good cross-fade, the algorithm described uses more sophisticated techniques to achieve proper cross-fading. The algorithm analyses the audio frequency-time spectograms of the two tracks to be cross-faded. This can be used to selectively suppress certain frequency components in the tracks such that current melodies seem to disappear and the next melody becomes more prominent. It can also filter out components of tracks that clash with each other, allowing for a smooth cross-fade [[#References|(Cliff, 2006)]].<br />
<br />
=User interface=<br />
<br />
=Pre-filter=<br />
<br />
=Excitement prediction with multiple regression=<br />
In order to come up with a useable product we need to narrow down the scope of this project. We decided to focus on engineering a proper pre-filter and feedforward system for track selection, based on the Spotify audio features. We mainly focus on the features "energy" and "valence". Even after extensive research, no formal definition or formula was found for the Spotify feature "energy". [https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ Spotify] itself describes it as a perceptual measure of intensity and activity based on dynamic range, perceived loudness, timbre, onset rate, and general entropy. It is a floating point number with a range between 0 and 1. The same holds for the "valence" feature; no formula can be found but Spotify describes it as a representation of the positivity or negativity of a track. It is a floating point number with a range of 0 to 1 where values close to 1 represent positive songs, whereas values close to 0 represent sad songs. <br />
<br />
Continuing, we rated 152 songs on "excitement" and then performed a multiple regression analysis with the Spotify features "energy" and "valence" to come up with a prediction formula for "excitement" based on these features. We rated "excitement" such that it represents how enthusiastic the song is, how excited one gets by listening to it. It is a floating point number ranging from 0 to 1. Values close to 0 represent songs that will not get people enthusiastic, whereas values close to 1 represent songs that are very exciting and fosters enthusiasm among listeners. Please not that for now the feature "excitement" is completely made up by ourselves and inherently subjective in nature. However, we deemed "excitement" as it is defined now a good parameter for track selection that makes the audience happy. We picked songs from three different genres of dance music: Techno, Hardstyle and Disco. We wanted to stick to dance music, but to diversify our research we considered three distinct genres. The results of the regression analyses are presented in the proceeding sections.<br />
<br />
==Multiple regression per genre==<br />
The first step in our analysis was to do a multiple regression for every distinct genre in our database - being either Techno, Hardstyle or Disco - to see whether the formula for "excitement" differs between genres and whether it gives any significant results to start with.<br />
<br />
===Multiple regression techno===<br />
The multiple regression model considering techno only turned out significant, R-squared = 0.1, F(2, 60) = 3.39, p = 0.04. Here, "excitement" was based on the Spotify features "energy" and "valence". It means that for techno the appropriate formula for excitement is the following: ex = 0.39 + 0.183*en + 0.208*v. Where "ex" is excitement, "en" is energy and "v" is valence. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.183<br />
| 0.116<br />
| 1.57<br />
| 0.121<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.208<br />
| 0.096<br />
| 2.11<br />
| 0.039<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.390<br />
| 0.100<br />
| 3.91<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression hardstyle===<br />
The model for hardstyle turned out non-significant, R-squared = 0.05, F(2, 37) = 0.91, p = 0.41. It means that with this sample of hardstyle songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| -0.171<br />
| 0.154<br />
| -1.11<br />
| 0.276<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.057<br />
| 0.063<br />
| -0.91<br />
| 0.371<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.800<br />
| 0.142<br />
| 5.61<br />
| 0.000<br />
|-<br />
|}<br />
<br />
===Multiple regression disco===<br />
The model for disco turned out non-significant, R-squared = 0.1, F(2, 46) = 2.55, p = 0.09. It means that with this sample of disco songs and their excitement ratings, no fitting formula is found by linear regression. The exact results are presented in the table.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.079<br />
| 0.109<br />
| 0.72<br />
| 0.474<br />
|-<br />
! scope="row"| <i>valence</i><br />
| -0.142<br />
| 0.077<br />
| -1.86<br />
| 0.070<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.693<br />
| 0.118<br />
| 5.86<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Multiple regression across genres==<br />
In the next step, we took all genres together and performed the same multiple regresion analysis on that dataset. This regression turned out significant, R-squared = 0.06, F(2, 149) = 4.99, p = 0.008. It means that there is a linear formula to predict "excitement" of a track based on the audio features "energy" and "valence" if you consider the three dance genres together. This formula is as follows: ex = 0.45 + 0.16*en + 0.07*v. If we generated this new predicted value from the formula, took the absolute difference between this predicted value and the "real" value for "excitement", the mean difference was 0.07 with a standard deviation of 0.06. All exact results can be found in the table below.<br />
<br />
{| class="wikitable", border="1" style="border-collapse:collapse"<br />
|-<br />
! scope="col"| <i>excitement</i><br />
! scope="col"| <b>Coefficient</b><br />
! scope="col"| <b>Standard Error</b><br />
! scope="col"| <b>t</b><br />
! scope="col"| <b>p</b><br />
|-<br />
! scope="row"| <i>energy</i><br />
| 0.160<br />
| 0.070<br />
| 2.28<br />
| 0.024<br />
|-<br />
! scope="row"| <i>valence</i><br />
| 0.070<br />
| 0.024<br />
| 2.93<br />
| 0.004<br />
|-<br />
! scope="row"| <i>constant</i><br />
| 0.450<br />
| 0.063<br />
| 7.13<br />
| 0.000<br />
|-<br />
|}<br />
<br />
==Discussion of the regression results==<br />
When we took each genre separately to come up with a prediction formula for "excitement", the results were not always promising. Only the regression for techno turned out significant. We think that this was mainly due to the small sample size (only 40, 49 and 63 songs used in every list) and the inherent subjective nature of the feature "excitement". Only one person rated this feature for every song making it a very subjective, personal variable. However, we should look beyond the results of this regression analysis only. In the future when our system is used at large music events, the variable "excitement" can be deduced in a much better way than by quantifying the subjective opinion of one person. One option could be to let every attender of a music festival rate about 10 relevant songs before attending the event. If 10,000 people do this the system can create 100,000 data points to base the regression on, instead of 152. One can imagine that this would give much better results than the provided analysis. Besides, when considering the regression analysis across genres we already came up with a model that has an average error of only 7%. One can imagine how small the error would become if 100,000 data points are used.<br />
<br />
Another option could be to generate the "excitement" information in a more physical sense. For example based on information gathered from heart rate monitors and/or brain activity sensors as used by de Effenaar <b>REFERENTIE NAAR UITWERKING GESPREK MET EFFENAAR</b>. This information is more quantitative in nature than people's subjective opinion on "excitement" level of a track. So, maybe this could function as input that generates a more robust formula for "excitement".<br />
<br />
Concluding, how the information on "excitement" is gathered is a point of discussion as well as an opportunity for improvement. What is most important is that a multiple regression model might be a good way to incorporate feedforward for track selection in our system. In that sense, the value of this particular research is not in the results of these regressions but in the method behind it.<br />
<br />
=Excitement matching with QET=<br />
<b>(STILL INCORPORATE FEEDBACK SENSORS SIMULATION?)</b><br />
<br />
For the excitement matching algorithm we want a module that takes in a QET from the user and transforms it into a mathematical formula, because a graph is what humans can work with, while a formula is what computers can work with. Because we want to answer to the user need of keeping the system simple and easy to use, we decided to let the user input the QET by means of "key points of excitement" in the music set. This is easier than drawing a whole graph. Thus, the user can specify at what points in time it desires certain values of excitement and the system does the rest. We created a MATLAB script to simulate such software and prove its working.<br />
<br />
==A script for QET matching algorithm==<br />
As an example, we chose as user input a 30 minutes music set, divided in 1 minute intervals. In the example case the user inputs the following desires: start the set at 0.7 excitement, after 5 minutes drop to 0.6 within 5 minutes, continuing climb to 0.8 within 10 minutes, then drop to 0.7 again within the next 5 minutes and finish of the set by climbing to the maximum excitement of 1.00 in the last 5 minutes. This is transformed to data points and depicted below.<br />
<br />
<br />
[[File:key_data_points.jpg|400px|Key data points from the user]]<br />
<br />
The next step is to interpolate for values between these key excitement points (in order to get a value for every minute of the set). We chose to use simple linear interpolation which gave decent results, however other sorts of interpolation are also possible and easily implemented using MATLAB. After interpolation, a polynomial fit is applied in order to come up with a mathematical formula for the QET. In this example a 7th order polynomial was created to describe the QET. The result is depicted below.<br />
<br />
[[File:interpolation_QET.jpg|400px|Interpolation result QET]]<br />
<br />
One can see that the result is quite promising. The polynomial curve follows the desired QET without an undesirable large error. The coefficients of the polynomial curve are also outputted by this script and can thus be used to construct a mathematical formula for the QET. This formula can then again be sampled (with a sampling period depending on the desired song length) to obtain the distinct excitement values the songs should have in time. In that way, the system can construct and output an ordering of the provided tracks based on this sampled formula for QET and the algorithm works. The MATLAB script is provided below.<br />
<br />
[[File:code_interpolation.jpg|MATLAB code]]<br />
<br />
=System overview (NEEDS EDITING)=<br />
<b>UNDER CONSTRUCTION?</b><br />
<br />
Below a graphic overview of the system is given. Important to note is that the grey area is the scope of our project; to design the rest is not feasible within our time budget. Besides, a lot of research is already done on the modules outside the grey region. Mixing tracks together perfectly is a desirable skill for all DJ's and therefore a skill for which a lot of DJ's seek help in the form of technology. Due to this high demand, a lot of properly working mixing algorithms already exist, for example [[#Algorithms for track selection|(Cliff, 2006)]]. Certain feedback sensors that measure excitement of the audience also already exist. A lot of wearable devices that measure all sorts of things like heart rate or motion already exist and have been used. One can look at one of the [[#Receiving feedback from the audience via technology|prior]] sections to gain information on this. Another real life example is [[#LINK EFFENAAR|(link naar uitwerking gesprek) De Effenaar]] that has used technology to measure brain activity and excitement among attenders of a music event. This lets us make the assumption that the mixing and the feedback part will work.<br />
<br />
Globally, the system works as follows. The system has a large database to pick music from. This database contains for every song an "excitement" value that is based on a [[#Excitement prediction with multiple regression|multiple regression]] output of the Spotify audio features "valence" and "energy". The music of this database is let through the [[#Pre-filter|pre-filter]]. The pre-filter filters the songs in the database on genre and SPOTIFY FEATURES (NOG BESLISSEN WELKE) based on the user input that is defined by the [[#User interface|user interface]]. In that sense, the pre-filter outputs an array of tracks that is already filtered to the user's desire. Next, the filtered tracks will be matched with the desired [[#Algorithms for track selection|QET]] (we will use an excitement graph instead of a tempo graph but the principle is the same). The result of this matching is a sequence of tracks that creates the playlist to be played for that evening (or morning?). This playlist is then fed to the [[#The algorithm|mixing algorithm]] to make sure that the system outputs a nicely mixed set of music to enjoy. This music is rated on excitement by means of [[#Receiving feedback from the audience via technology|feedback sensors]]. This feedback is used to update the playlist via the excitement matching module. Thus, if the audience is not happy with the currently playing music the system can act upon that.<br />
<br />
[[File:system_overview_graphic.JPG|none|800px|System overview]]<br />
<br />
=References=<br />
<br/><br />
Atherton, W. E., Becker, D. O., McLean, J. G., Merkin, A. E., & Rhoades, D. B. (2008). U.S. Patent Application No. 11/466,176.<br />
<br />
<br/><br />
Barkhuus, L., & Jørgensen, T. (2008). Engaging the crowd: studies of audience-performer interaction. In CHI'08 extended abstracts on Human factors in computing systems (pp. 2925-2930).<br />
<br />
<br/><br />
Berkers, P., & Michael, J. (2017). Just what makes today’s music festivals so appealing?.<br />
<br />
<br/><br />
Choi, K., Cho, K. <i>“Deep Unsupervised Drum Transcription”</i>, 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands, 2019.<br />
<br />
<br/><br />
Choi, K., Fazekas, G., Cho, K., & Sandler, M. (2017). <i>A tutorial on deep learning for music information retrieval</i>. arXiv preprint arXiv:1709.04396.<br />
<br />
<br/><br />
Cliff, D. (2006). hpDJ: An automated DJ with floorshow feedback. In Consuming Music Together (pp. 241-264). Springer, Dordrecht.<br />
<br />
<br/><br />
De León, P. J. P., & Inesta, J. M. (2007). <i>Pattern recognition approach for music style identification using shallow statistical descriptors</i>. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(2), 248-257.<br />
<br />
<br/><br />
Feldmeier, M. C. (2003). Large group musical interaction using disposable wireless motion sensors (Doctoral dissertation, Massachusetts Institute of Technology).<br />
<br />
<br/><br />
Feldmeier, M., & Paradiso, J. A. (2004, April). Giveaway wireless sensors for large-group interaction. In CHI'04 Extended Abstracts on Human Factors in Computing Systems (pp. 1291-1292).<br />
<br />
<br/><br />
Freeman, J. (2005) <i>Large Audience Participation, Technology, and Orchestral Performance</i> in Proceedings of the International Computer Music Conference, 2005, pp. 757–760.<br />
<br />
<br/><br />
Gates, C., & Subramanian, S. (2006). A Lens on Technology’s Potential Roles for Facilitating Interactivity and Awareness in Nightclub. University of Saskatchewan: Saskatoon, Canada.<br />
<br />
<br/><br />
Gates, C., Subramanian, S., & Gutwin, C. (2006, June). DJs' perspectives on interaction and awareness in nightclubs. In Proceedings of the 6th conference on Designing Interactive systems (pp. 70-79).<br />
<br />
<br/><br />
Greasley, A. E. (2017). Commentary on: Solberg and Jensenius (2016) Investigation of intersubjectively embodied experience in a controlled electronic dance music setting. Empirical Musicology Review, 11(3-4), 319-323.<br />
<br />
<br/><br />
Humphrey, E.J., Durand, S., McFee, B. <i>“OpenMIC-2018: An open dataset for multiple instrument recognition”</i>, 19th International Society for Music Information Retrieval Conference, Paris, France, 2018.<br />
<br />
<br/><br />
Hamel, P., & Eck, D. (2010, August). <i>Learning features from music audio with deep belief networks</i>. In ISMIR (Vol. 10, pp. 339-344).<br />
<br />
<br/><br />
Hödl, Oliver; Fitzpatrick, Geraldine; Kayali, Fares and Holland, Simon (2017). <i>Design Implications for TechnologyMediated Audience Participation in Live Music</i>. In: Proceedings of the 14th Sound and Music Computing Conference,<br />
July 5-8 2017, Aalto University, Espoo, Finland pp. 28–34.<br />
<br />
<br/><br />
Hoffman, G., & Weinberg, G. (2010). <i>Interactive Jamming with Shimon: A Social Robotic Musician</i>. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, 3097–3102.<br />
<br />
<br/><br />
Huron, D. (2002). <i>Music information processing using the Humdrum toolkit: Concepts, examples, and lessons</i>. Computer Music Journal, 26(2), 11-26.<br />
<br />
<br/><br />
Jannach, D., Kamehkhosh, I., & Lerche, L. (2017, April). <i>Leveraging multi-dimensional user models for personalized next-track music recommendation</i>. In Proceedings of the Symposium on Applied Computing (pp. 1635-1642).<br />
<br />
<br/><br />
Johnson, D. (n.a.) <i>Robot DJ Used By Nightclub Replaces Resident DJs</i>. Retrieved on 09-02-2020 from http://www.edmnightlife.com/robot-dj-used-by-nightclub-replaces-resident-djs/ <br />
<br />
<br/><br />
Kashino, K., Nakadai, K., Kinoshita, T., & Tanaka, H. (1995). <i>Application of Bayesian probability network to music scene analysis</i>. Computational auditory scene analysis, 1(998), 1-15.<br />
<br />
<br/><br />
McAllister, G., Alcorn, M., Strain, P. (2004) <i>Interactive Performance with Wireless PDAs</i> Proceedings of the International Computer Music Conference, 2004, pp. 1–4.<br />
<br />
<br/> <br />
Pasick, A. (21 December 2015) <i>The magic that makes Spotify's Discover Weekly playlists so damn good</i>. Retrieved on 09-02-2020 from https://qz.com/571007/the-magic-that-makes-spotifys-discover-weekly-playlists-so-damn-good/ <br />
<br />
<br/><br />
Pérez-Marcos, J., & Batista, V. L. (2017, June). <i>Recommender system based on collaborative filtering for spotify’s users.</i>In International Conference on Practical Applications of Agents and Multi-Agent Systems (pp. 214-220). Springer, Cham.<br />
<br />
<br/><br />
Shmulevich, I., & Povel, D. J. (1998, December). <i>Rhythm complexity measures for music pattern recognition</i>. In 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No. 98EX175) (pp. 167-172). IEEE.<br />
<br />
<br/><br />
Wen, R., Chen, K., Xu, K., Zhang, Y., & Wu, J. (2019, July). <i>Music Main Melody Extraction by An Interval Pattern Recognition Algorithm</i>. In 2019 Chinese Control Conference (CCC) (pp. 7728-7733). IEEE.<br />
<br />
<br/><br />
Yoshii, K., Nakadai, K., Torii, T., Hasegawa, Y., Tsujino, H., Komatani, K., Ogata, T. & Okuno, H. G. (2007, October). <i>A biped robot that keeps steps in time with musical beats while listening to music with its own ears</i>. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1743-1750). IEEE.<br />
<br />
<br/><br />
Zhang, L., Wu, Y., & Barthet, M. (ter perse). <i>A Web Application for Audience Participation in Live Music Performance: The Open Symphony Use Case</i>. NIME. Geraadpleegd van https://core.ac.uk/reader/77040676</div>S165296