PRE2016 3 Groep10

Advanced Elderly Emergency System (A.E.E.S.)

This is the wiki page of group 10 of the USE course: Project Robots Everywhere at the Eindhoven University of Technology. Here we will thoroughly describe all aspects of our project, in which we design the AI part of a wearable fall-detecting device for elderly people.

Design team

We have a diverse team of 6 people from different majors of the TU/e:

Name	Student ID	Department
Lennard Buijs	0959903	Mechanical Engineering
Bram Grooten	0885158	Applied Mathematics
Ken Hommen	0911594	Industrial Engineering
Pieter van Loon	0861532	Software Science
Steef Reijntjes	0944701	Electrical Engineering
Man-Hing Wong	0944285	Electrical Engineering

Problem statement

Problem

In the Netherlands we have more and more elderly people. For them it can be more difficult to balance themselves when standing or walking, so it regularly happens that they fall. When they do, they often have trouble standing up. Also, they have a higher chance of injuries, with hip fractures being the most common. The problem especially arises if these elderly are severely injured by a fall, and therefore can't get up to reach for help.

Solution

A device that should be worn by, mostly, elderly that detects when one has fallen. It can automatically send a warning to ‘ICE’-persons or even call 112. Automatically sending its location along with it. A microphone and camera can in this case be used to observe the situation even faster. By connecting the device to the internet, this all can be made possible even faster. Also the device can ‘ask’ questions to the owner in case of emergency, which can be answered by simply talking back.

Objectives

To effectively and successfully end this project, a list of objectives is created to ensure weekly improvements, which will help in obtaining a final product. List of objectives:

• Research state-of-the-art technology
• Establish list of requirements
• Design the system that follows every requirement, this would be our ideal deliverable
• Create a final presentation, explaining our design

USE aspects

In the section below, all needs and demands of the users, society and entreprise (USE) has been listed and discussed. The needs and demands that play a role in the final design are specifically indicated.

User

In particular, the users of the Advanced Elderly Emergency System are the elderly. Since the number of elderly is increasing, more elderly people will need a way of calling for help when an emergency occurs. Nowadays, an elder may have an arm wrist or necklace which has a button to call for help. However, in the future, there will be too many elderly for the number of nurses or doctors. By asking questions autonomously, the doctor's job will be easier, which means it will also correspondingly help the enterprise. The chance of getting helped after getting injured from falling will be significantly greater.

Primary users

Elderly that regularly experience falling.

• The system should not negatively influence physical actions performed by the elderly in any way.
• The user should be able to manually press the button for a certain amount of time when help is needed.
• The user should be able to set a time, before a medical instance will be warned, in case when no response can be sensed by the system.
• Data information what has been collected and transmitted should be secure and private.
• The system should give relevant, understandable questions to the elderly and respond morally correct.
• The system's volume can be changed in a preferable way.
• The system should notify the elder on time when the system is not working properly i.e. due to low battery.

Secondary users

The secondary users are the next of kin of the elderly (i.e. the people who are living together with the elderly, or are responsible for their care).

Tertiary users

Medical health instances that should stay alert all the time and can provide help in emergency cases.

• Geolocation data and other information what can be obtained by the system should be held private. Only these trained medical health instances and the relevant person himself may have a look insight.
• Information what has been sent to the local emergency instances should be accurate.
• Accidentally calling the health instances by pressing on the button should be prevented and not occur.

Society

Due to time managing reasons, the society group of USE has not been discussed and analyzed. It has not been involved during the entire period of time of the project.

Entreprise

The government acts as the entreprise, since the government has to take care of the health insurances of society.

Responsibility

Who should be responsible when the wearable system does not operate properly in critical situations?

Interview with elderly

On Friday, March 10th we went to Vitalis Berckelhof, an elderly home in Eindhoven, to interview the elderly about their thoughts on our idea.

Research

Medical Alert Systems

Currently there a few different wearable emergency devices for elderly. All of them have slightly different functions. Firstly, there is the Medical Guardian. The Medical Guardian is considered the best medical alert system available. The Medical Guardian (Premium) is a medical alert system that can be worn as an arm wrist, belt or necklace and has the following features:

• Fall detection that will call for help when the patient has fallen
• A button that will call for help and contacts you with an employee of Medical Guardian
• GPS-tracking
• Heat-sensor in case of fire

Other devices and further explanation on the Medical Guardian can be found further in this chapter.

Fall detection

The fall detection is most important to this research, since it is one of the main features of the Advanced Elderly Emergency System. Currently, most fall detection systems utilise accelerometers and gyroscopes. With these sensors the movement of the patient can be measured. A sudden change of direction can then be also be measured and calculated.

Most sensors have a certain waiting time before calling for help, because it checks for movement after a possible fall. If the patient moves, the system will not call for help. When an AI is added to the system, this waiting time can be reduced a lot, because the AI can ask the patient whether he or she has fallen.

In the A.E.E.S. the fall control system by F. Wu. et al. will be used. This system requires only a single triaxial accelerometer and therefore uses significantly less power than the systems with gyroscopes. This means the elderly people will have to charge their alert system less. More on this fall detection system and how it works is explained in chapter 6.4 "Implementation of fall detection sensor".

Speech recognition

Speech recognition is another important feature of the Advanced Elderly Emergency System. There are many applications for speech recognition. A very wellknown software is Siri. Siri is able to recognise one's voice and hear what the person is asking.

Current devices

ZEMBRO

“Zembro bracelet is the first intelligent bracelet specially designed for the elderly. It looks good. And better yet: it’s good company. It offers you the freedom, independence and peace of mind that is so crucially important.” An elegant bracelet, it features smart functions and is connected to an app. Zembro personal alarm keeps you in touch with each other. At any time. Including those moments when it is so crucially important. And that gives everyone peace of mind. Because you’re always nearby. Even when you’re not in the neighbourhood. The Zembro bracelet makes the connection with your family or friends even stronger. And isn’t that the most beautiful gift that you could possibly give someone?

LIMMEX

Limmex in general:

The watch is always within reach. It works outdoors just as reliably as it does at home. Simply press one button and the Limmex Emergency Watch will set up a telephone call for you. Thanks to the built-in loudspeaker and microphone you can directly speak to your person(s) selected. You define which telephone numbers should be called. In an emergency situation every minute counts. Communicating with people in an emergency can be fraught with problems and cost valuable time and delay from the rescue services. All models come with an integrated high performance GPS-module (A-GPS). Thanks to the GPS-function the Limmex wearers can quickly pinpoint their location via satellite when they are outside of a building.

The Limmex Watch also sends an SMS to the person called after the alarm has been raised. The SMS contains a link to a map indicating the location of the watch at the time of the emergency call. You can also email the map link to anybody you wish. Your watch will only be located after an emergency call has been made. This means that the Limmex Watch battery lasts longer and that you decide when you want it to be located. If there is no GPS signal reception (e.g. inside buildings), GSM tracking is used over the mobile phone network.

Limmex for elderly:

Most elderly persons would like to live for as long as possible in their own home. A great number of them still have good health but nevertheless realize that the risk of falling, feeling unwell, dizziness or suffering from other health problems increases substantially with age. Many of them would like to own a device which offers them security without limiting their individual lifestyles. Traditional domestic emergency call systems only partially fulfill the requirements of this target group. Monitoring is only guaranteed inside the home – this is in spite of the fact that many older people remain mobile well into old age. In addition, traditional emergency call systems are complicated to operate because of their costly installation and their great number of technical alarms – and their design is unsatisfactory. With Limmex it is possible to solve most of these problems: the product offers security also outside the home, it is technically considerably easier than current solutions and due to the convincing watch collection it meets even high demands on design and aesthetics.

MEDICAL GUARDIAN

Get Help in 3 Simple Steps

Step 1

Press the button on your Medical Guardian Pendant. The small pendant can be attached to a belt, worn around the neck, or around the wrist. While you are home, the device should be worn at all times. Because it is 100% waterproof, the pendant can be worn in the shower or bath. When pressed, the device sends a wireless signal to the device's base station, in your home. This base station alerts the monitoring centre of an emergency.

Step 2

Communicate with a certified medical operator. Within moments of pressing the button on your Medical Guardian you will be contacted, via a two-way radio in the device, to a certified and trained emergency responder. The operator will assess your situation and help you determine what type of help you need. Able to access your file, the responder can view your medical profile, medications, preferred doctors, local hospital, and contact information for friends and family members. This means you don’t have to worry about an ambulance being sent to your home anytime you use the Medical Guardian system. You can arrange to have a neighbour come over, have a family member called, or anyone else you designate as a contact. If you are unable to respond, operators will send local emergency personnel to your home.

Step 3

Help is on the way! Completely portable, the Medical Guardian medical alert system can go anywhere you do. You can bring the security and peace-of-mind of Medical Guardian with you when you are on vacation, visiting family, or even permanently moving within the United States. Just plug the base station into the wall, notify the monitoring centre of your new location, and you are done. Medical Guardian provides service in all 50 states, allowing you to travel with confidence.

Comparison

Here will be explained why the A.E.E.S. is better than just using an emergency button, and show what it really adds in new functionality.

Approach

The A.E.E.S improves on the state of the art technology by implementing an AI into the system which helps the user in case of an emergency where the user is unable to manually activate the needed functions on the device. This AI determines whether or not certain actions need to be taken depending on the current situation/enivornment. These actions include, but are not limited to:

• Calling emergency services
• Contacting family members/friends
• Asking the user what his/her state is

Requirements

These are components that each part of our design will most likely need.

The fall-detection-device

• Wearable, for example one of the following:
  • Belt clip
  • Wrist band
  • Necklace
• Sensors to detect falling:
  • accelerometer
  • barometer
  • gyroscope
• Air quality measuring sensor

The AI (smartwatch app)

• Microphone, for speech recognition
• Speakers, so it can speak to the user
• GPS to track location
• Internet and phone connections
  • Call emergency services
  • Contact next of kin
  • Share location with next of kin
• Button to manually trigger emergency state

Design options

These are the possible options of designs that would could produce. The fall-detecting-device attached to one's belt will detect the fall. This belt clip could:

• have its own microphone and speaker to communicate with the user.

• or the belt clip could be connected to a smartwatch app. This smartwatch will then need to have a microphone and a speaker. Many smartwatches exist which don't have a speaker, thus we would need to look for a specific watch that does have one.

• or it could connect with a smartphone app. This is simpler in design for us, and most smartphones both have a microphone and speakers. However, many elderly won't have their smartphone with them at all times, if they even have a smartphone.

• or we could make our own wrist band with a built-in microphone and speaker to communicate with the user.

It seems like we should definitely make the belt clip, and see what we can add because of our limited time. But, the functionality of this belt device alone already exist, as we have seen in our research. To really add something to the state-of-the-art, we need to combine it with some form of AI. Thus another option is starting with the AI functionality.

We have decided to go for the third option, thus we will make a smartphone app. We actually think the design with the smartwatch is the best idea, because of the wearable functionality. Elderly don't always have a smartphone with them. But since none of us have a smartwatch, we will design a smartphone app with the same behaviour that the smartwatch app would have. We won't be making the actual fall-detecting-device, because as we said earlier, this already exists. It is not yet implemented into society, though.

Figure XXX: The four possible design options.

AI functionality

The AI in the device is activated when the device registers a falling motion, or when the user presses the action button. This AI will then immediately contact emergency services to inform them that an elderly person fell. The emergency services then know the place/situation of the elderly person, and they will act accordingly by for example sending out an ambulance. After this, the AI will ask the person if they are okay, and eventually if the AI should contact family/friends. If the AI does not receive feedback from the user, the AI will send a message to family or friends that something is wrong with the user.

Specific questions from the AI

Suppose a fall has been detected (or the user has manually pressed the button). Then our device immediately triggers the AI and it will ask:

"Are you okay?" (or something in this trend)

• If the person answers "yes", we ask one question again, just to make sure. Like: "Do I need to call emercency services?" The answer will then decide if we take all the next steps of calling emercency services and the next of kin, or just returning to normal state again. In a machine learning environment, even when the answer to the second question is no, it still is valuable feedback to our system. Our AI could learn in what ways it detects a fall, and which are most of the time not severe.

• If the person answers "no", we notify emergency services immediately, while going on with the conversation with the elder. We will ask questions like: "Do I need to call emergency services?" and if the answer is "no", another "Are you sure?" will follow. If the person is sure, we will cancel the emergency call.

• If the person doesn't answer within 10 seconds, we will notify emercency services. After 3 seconds, for example, we will repeat the first question a bit louder. The elderly might not have heard it. When there is still no response, we could ask a different question after 6 seconds, like: "Do I need to call emergency services?"

When we have made sure that the elder needs help, we thus contact emergency services, like an ambulance. Then we send our location details

Estimating the seriousness

The answers that an elderly person gives might not be very clear. Therefore, the AI should be able to figure out which questions it needs to ask to get the best information. For example, in a case from the company CSIservice, they once received an alarm from a woman saying she had a stomach ache. At first, people might underestimate the situation when hearing "stomach ache." But because the employers of the alarm centre are trained to keep asking questions, they discovered how serious the situation was. An ambulance was called, and it turned out that the woman had intestinal aneurysm.

Implementation of fall detection sensor

Design by Falin Wu et al.

The fall detection sensor in the A.E.E.S. will be fully based on the fall detection sensor proposed in "Development of a Wearable-Sensor-Based Fall Detection System" by F. Wu et al. The advantages of using this sensor are:

• Can be worn on the waist, but does not have to be fixed in position
• Only uses a single triaxial accelerometer for fall detection
• Is power efficient, making the elderly able to travel without constantly having to recharge it.

This design is an acceleration threshold based design, which will be further explained in Chapter 6.4.2 "Fall Detection Algorithm". The reliability of this sensor is fairly high. In Table XXX the test results of the system test are shown. As you can see, the sensor detects a fall 220 out of 240 times, resulting in an average sensitivity of 91,7%. However, also notable is the number of false alarms when a person is jumping, walking or resting. Most elderly people that need a medical alert system will not regularly be jumping, but to limit the number of false alarms it is necessary to experiment with the number of a_threshold.

Table XXX: Test results of the design with different motion types

Fall Detection Algorithm

The fall detection algorithm consists of five states. The state diagram is the same as the original design by F. Wu et al., with the exception of state 5. The state diagram can be found in figure XXX.

State 1: Initial state

The acceleration a of the elder will be measured by the triaxial accelerometer. This acceleration will consist of a_x, a_y and a_z. The total acceleration can be calculated with the formula:

|a| = sqrt(a_x² + a_y² + a_z²)

g_before will be set to the current acceleration.

State 2: Acceleration Acquisition state

In this state the program will check if the current acceleration is higher than the set number a_threshold. It is necessary to do some experiments to find the right number for this a_threshold. When this acceleration is reached, the program will progress to state 3.

State 3: Impact state

When a person falls, it will reach a state of impact, after which he or she will be laying still. At the point where the person is laying still, the acceleration picked up by the accelerometer will equal g. Since it takes a while for a person to finish their falling, a t_threshold is necessary to minimize false alarms. This time will be set to at least two seconds. When the person has come to a stop and two seconds have passed, the program will progress to state 4.

State 4: Lying state

In the lying state, the system checks whether the person that has fallen has changed its angle relative to the ground. When the person starts by standing, he or she will have a rotation angle of 90 degrees after the fall. The rotation angle can be calculated by knowing the direction of the gravity g_before and the direction after g_after. g_after will be set to the acceleration after the fall, and the rotation angle can then be calculated using the formula:

θ = 2 arctan() (insert picture for formulas)

When the rotation angle is approximately 90 degrees, the program will progress to state 5.

State 5: AI state

In this state, the AI with speech recognition takes over. When the AI has done its job, the program will go back to state 1.

Figure XXX: State diagram of fall detection algorithm

Hardware Analysis

Measurement hardware used in this system include a motion sensor and a GPS and GSM service unit. Together with a microprocessor for processing data and a speaker/microphone set for human interaction, these units comprise the entire system which will be powered by a low-voltage battery (1200 mAh with 3.7 V is enough to last a couple of days).

• An ADXL345 can be used as motion sensor and will typically cost €3.50.
• A SIM808 GPS GSM GPRS which is used for GPS and GSM service which costs around €40.00.
• A TI MCU MSP430F1611 microprocessor will control the whole system and imply the detection algorithm. This costs approximately €17.00.

When looking at ways to implement a smartphone/smartwatch, an extra bluetooth module has to be added to establish a bluetooth connection between said device and the microprocessor. This means that the GPS and GSM service is no longer necessary and allow for a cheaper system.

This requires a different microprocessor and perhaps a bigger battery to support full functionality. Other possibilities include making an internet connection between the smart device and the fall detection system. This also requires a module to establish a Wi-Fi connection or an entirely different processor which has integrated Wi-Fi functionalities.

Final design

Design description

Our final design is a wearable system, which can mostly be worn by elderly. This system consists of two parts. Those parts are each individually responsibly for different functionalities of the whole system. The lower part, attached on the belt, is responsible for fall detection, while the upper part, attached as arm wrist, should be held responsible for the speech recognition.

For determining the spot on the human body to put the lower part on, we had to look for a location what is most balanced on each time sequence. It had to be a place, what did not change much from its location when performing human actions. This part will be the main part, which is relatively bigger in size than the upper part on the arm wrist due the fact that it contains the microprocessor. The upper part is not suited for this functionality, since that would be less comfortable at the end.

The upper part is an arm wrist fulfulling the speech recognition task. Artificial intelligence should be held into consideration when designing this part. The arm wrist should ask questions to the relevant person that just fell. Artificial intelligence can be used to have a conversation with the person based on the situation, and further on determine the state of the current situation. Based on the state of this situation, it can perform actions fully autonomous by notifying health instances in cases when needed. This can speed up the process to help the wounded elderly, or even save their lives.

Possible expansions

These are all expansions to the project, which would greatly improve the A.E.E.S. Unfortunately, we didn't get to these topics, or decided not to because of their complexity.

Detecting tone of voice

By analysing the tone of the users voice it should be possible to make some assumptions about the severity of the situation.

Estimating the seriousness

With this detection, and using other methods like thoroughly investigating the situation with many questions, the AI could make a better estimate of the seriousness of the situation.

Learning users patterns

The system could learn from when an user often triggers the system without there being an emergency. It can for instance update it's thresholds or also take time of day and or the users patterns into consideration.

Symptom analysis

The system can greatly be used for injuries caused by falling, but as an extension, an additional data package with all information and symptoms of diseases can be added to the artificial intelligent machine. On this way, the intelligent system can also be used for injuries caused by health problems. The system interprete the symptoms and checks whether a great minority of the proposed symptoms corresponds with the information of a certain disease. The artificial intelligent agent can perform a diagnosis and gives an idea of what the disease can be, and further on give a recommendation of what the person can do best and eventually call an emergency service. For example, it can happen that an elder makes use of the system because of stomache ache. Usually, the system asks questions to give an expression of the seriousness of the situation, but with the introduction of a data package with all disease symptoms, it can ask for symptoms explicitly to determine the situation and health even better and more accurately.

Literature

Other pages with important information on this subject:

URLs

Medical guardian emergency system

https://www.medicalguardian.com/product/premium-guardian

The Best Medical Alert Systems of 2017

http://www.toptenreviews.com/health/senior-care/best-medical-alert-systems/

Speech recognition library

http://arjo129.github.io/uSpeech/

Appendices

This section gives an overview of the progression and planning of our design project. This mainly concerns organizational as well as technical tasks, decisions and ideas that have been performed within our project environment to maintain a successfull and appropriate end result. Explanations on how certain decisions has been made by our design team can be found in the following, relevant sections:

Appendix A

Project progress (log): Gives an overview of the general and specific tasks for each member every week.

Appendix B

Planning: A schedule of the general tasks.