PRE2022 3 Group8

From Control Systems Technology Group
Jump to navigation Jump to search

MAIL

Dear team,

Group 8:

Fee Noordermeer (1587404) - [[1]]

Matthieu den Otter (0333626) – [[2]]

Merel Nienhuis (1721763) - [[3]]

Sam Lancee (1640364) - [[4]]

Wilco van der Veen (1021299) - [[5]] Automotive Technology

Dave Hannet (1715666)


Subject: A wearable device for people with problems remembering names (mild dementia, temporal or permanent memory loss or just bad memory) that recognizes a face and gives the user a corresponding name via a Bluetooth connect earplug (possibly hearing device).  

Special request: one of our team members has classes during hr 1-2 on monday. Is it possible to arrange a meeting during hr 3 or 4?

Kind regards, group 8



Fee Noordermeer, Matthieu den Otter, Merel Nienhuis, Sam Lancee and Wilco van der Veen


Problem statement


Objective


Who are the user?

The user consists of people that have problems with memory loss. This is typically a group of people of older age. The group consists of for example people with dementia.


What do they require?


Approach


Milestones


Deliverables


Who’s doing what?

- hardware (Wilco)

- software (Mathieu, Merel, Sam)

  • facial recognition
  • Audio
  • Data base

- User writing part (Merel, Sam)

-


State of the Art (25 articles)

Sam

1 https://dl.acm.org/doi/abs/10.1145/1090785.1090806?casa_token=M2Z7T1CRUDAAAAAA:OTdONvxGmFEzED-EVnPLnwj3W6lwWpdxyHNIHEX9YsEzU3mSZbWlq99lttpeHd4piY9kanAwNNJ4 --> pretty much what we plan on doing, but for people with visual impairments.  

2 https://dl.acm.org/doi/abs/10.1145/2459236.2459262?casa_token=ujZ8wpZTorkAAAAA:WDsDXJZZCHIuySAv53DFSyvW5xGCtSdaq8riNw07sxXSfXgo8CjtEjNuURSYpaDu6vGnEVOlw4I8 --> what we are doing but for a different user group

3 https://link.springer.com/chapter/10.1007/978-3-319-16634-6_31 --> facial recognition in your glasses, with user interface desplay

4 https://link.springer.com/chapter/10.1007/978-3-319-92384-0_11 --> wearables and security/privacy problems

5 https://journals.sagepub.com/doi/pdf/10.1177/1533317519883493 --> looking at the acceptence of using wearables for person with dementia, from caregiver view

Wilco

1 https://ojs.wiserpub.com/index.php/AIE/article/view/62/328 --> An Innovative Approach for Face Recognition Using Raspberry Pi

2 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3513528 --> Data Protection Challenges in the Internet of Things Era: An Assessment of Protection Offered by PDPA 2010

3 https://www.sciencedirect.com/science/article/pii/S0747563217304284 --> Consumer adaptation and infusion of wearable devices for healthcare

4 https://academic.oup.com/ageing/article/30/6/455/39799?login=true --> Assistive technology in elderly care

5 https://www.mdpi.com/1648-9144/56/6/257 --> Stress in the Volunteer Caregiver: Human-Centric Technology Can Support Both Caregivers and People with Dementia

Merel

1 https://www.faceblind.org/research/ --> what is prosopagnosia (official name for having problems with recognizing faces) + interesting sentence for our project: " It is important to note that prosopagnosia is defined by problems recognizing faces, not recalling names. "

2 https://www.cbc.ca/documentaries/the-nature-of-things/this-woman-can-t-recognize-her-own-daughter-s-face-but-technology-is-helping-her-1.6312174 --> zelfde maar dan met app op telefoon

3 https://www.researchgate.net/publication/263091789_A_room_full_of_strangers_every_day_The_psychosocial_impact_of_developmental_prosopagnosia_on_children_and_their_families --> impact on families, important to manage to help them with face recognition difficulties

4 https://courses.engr.illinois.edu/ece445/getfile.asp?id=12216 --> requirements for device that is similar to our idea

5 http://wearcam.org/smc2020/IEEE_SMC2020_0700_prosopagnosia.pdf --> glasses, but also proof that participants have shown improvements after using glasses

Matthieu

1 https://www.orcam.com/en/myeye2/#:~:text=OrCam%20MyEye%20is%20a%20revolutionary,live%20a%20more%20independent%20life! Currently existing device that does all and more, but high costs

2 https://nitratine.net/blog/post/python-face-recognition-tutorial/ Face recognition with python

3 https://www.sciencedirect.com/science/article/pii/S0197457214001360?via%3Dihub Maybe useful to set gradations in dementia

4 https://dl.acm.org/doi/pdf/10.1145/3351248 Face Recognition Assistant for People with Visual Impairments

5 https://aging.jmir.org/2019/1/e13378/PDFA Web-Based Mobile App With a Smartwatch to Support Social Engagement in Persons With Memory Loss: Pilot Randomized Controlled Trial

Fee

1 Support vector machines for face recognition

2 Training Support Vector Machines: an Application to Face Detection

3 Robust face detection using local CNN and SVM based on kernel combination

4 Computationally Efficient Face Detection

5 Support vector machine based multi-view face detection and recognition


Summaries articles 0LAUK0 Merel

Article 1

https://www.faceblind.org/research/

Understanding Prosopagnosia

People who suffer from prosopagnosia have difficulties with the recognition of facial identity. This can create social problems. Prosopagnosics do not have problems with recalling names.

It is possible that prosopagnosia occurs after brain damage due to trauma, stroke or a disease. Sometimes, it is the case that this impairment has always been there for someone.

Article 2

https://www.cbc.ca/documentaries/the-nature-of-things/this-woman-can-t-recognize-her-own-daughter-s-face-but-technology-is-helping-her-1.6312174

This woman can't recognize her own daughter's face, but technology is helping her

This article is about a woman that is not able to recognize faces, but technology does help her.

The woman tells that her inability to recognize faces can cause issues. She feels insecure and people are hurt that she does not recognize them. She does recognize voices and other things like birthmarks. Sometimes she does not even recognize herself.

A personalize app was developed for her to help her recognize people. People stand on the opposite of the phone, the camera scans the face and gives the name which can be seen by the woman.

The reactions are positive, for the woman with prosopagnosia but also for her close ones. They are delighted that the woman did recognize her with the help of the phone. The woman felt sure which felt good for her, it was not just a guess. However, the woman was not pleased about the fact that she had to look at the phone the whole time to be able to recognize someone.

Article 3

https://www.researchgate.net/publication/263091789_A_room_full_of_strangers_every_day_The_psychosocial_impact_of_developmental_prosopagnosia_on_children_and_their_families

“A room full of strangers every day”: The psychosocial impact of developmental prosopagnosia on children and their families

Individuals with developmental prosopagnosia have severe face recognition difficulties due to a failure to develop the necessary visual mechanisms for recognizing faces. These difficulties occur in the absence of brain damage and despite normal low-level vision and intellect. Adults with developmental prosopagnosia report serious personal and emotional consequences from their inability to recognize faces, but little is known about the psychosocial consequences in childhood. Given the importance of face recognition in daily life, and the potential for unique social consequences of impaired face recognition in childhood, we sought to evaluate the impact of developmental prosopagnosia on children and their families. We conducted interviews with 8 children with developmental prosopagnosia and their parents. A battery of face recognition tests was used to confirm the face recognition impairment reported by the parents of each child. We used thematic analysis to develop common themes among the psychosocial experiences of children with developmental prosopagnosia and their parents. Our findings indicate a need for increased awareness and treatment of developmental prosopagnosia to help these children manage their face recognition difficulties and to promote their social and emotional wellbeing.

Article 4

https://courses.engr.illinois.edu/ece445/getfile.asp?id=12216

Assistive Technology for Patients with Medical Face Blindness

This article gives an overview of requirements and verification for a device that assists social interaction for individuals with prosopagnosia. It also states that it is important to follow the IEEE Code of ethics and to develop something that is aware of potential safety-related issues.

Article 5

http://wearcam.org/smc2020/IEEE_SMC2020_0700_prosopagnosia.pdf

Face Recognition and Rehabilitation: A Wearable Assistive and Training System for Prosopagnosia

The design and implementation of an integrated wearable face recognition and training system for prosopagnosia patients are presented. The purpose of this assistive technology is to provide real-time memory assistance and long-term rehabilitation. The real-time face recognition mode provides audio and visual notification of people who interact with the subject, while the at-home training mode combines features of mnemonic and perceptual training to help with prosopagnosia rehabilitation. In addition, a custom eye tracker is developed to determine the person whom the subject is making eye contact with within a crowd. Using the inverted face effect to mimic the difficulties of prosopagnosia patients, clinically healthy participants have shown improvements in their face-naming abilities. Early results indicate the system’s potential to enrich the well-being of prosopagnosia patients.

Summaries articles 0LAUK0 Matthieu

Article 1 https://www.orcam.com/en/myeye2/#:~:text=OrCam%20MyEye%20is%20a%20revolutionary,live%20a%20more%20independent%20life!

OrCam MyEye is a voice activated device that attaches to virtually any glasses. It can instantly read to you text from a book, smartphone screen or any other surface, recognize faces, help you shop on your own, work more efficiently, and live a more independent life! OrCam MyEye conveys visual information audibly, in real-time and offline. COST: >3000 euros at lowvisionshop.nl

Article 2 https://nitratine.net/blog/post/python-face-recognition-tutorial/

Using the face_recognition for detecting and matching a face

Article 3 https://www.sciencedirect.com/science/article/pii/S0197457214001360?via%3Dihub

Dementia can be divided in stages. These can be useful to specify target group for device

Article 4 https://dl.acm.org/doi/pdf/10.1145/3351248 Face Recognition Assistant for People with Visual Impairments

Gives interesting background for investigating the requirements and challenges that must be addressed in the design of a face recognition system for helping VIPs recognize people with whom they have weak-ties
Article 5 https://aging.jmir.org/2019/1/e13378/PDFA Web-Based Mobile App With a Smartwatch to Support Social Engagement in Persons With Memory Loss: Pilot Randomized Controlled Trial

The Social Support Aid (SSA) App on mobile phone: Reasons why presented technology was not usefull: (1) complexity of the SSA, (2) enrollment process, (3) impracticality, (4) stigma, and (5) functionality of the SSA.

Summary articles Sam

Article 1

A wearable face recognition system for individuals with visual impairments

Icare interaction assistant which is there for helping with social interactions for people who are visually impaired. Talks about the problems with using real-time facial recognition for a wearable device. This device is their to see what the other person for facial expressions has.

Article 2

Who are you?: A wearable face recognition system to support human memory

In this paper they talk about the use of a wearable that is there to aid human memory. This can be for remembering names of other people. They first deal with different design ideas and after that a face recognition approach.

Article 3

A Wearable Face Recognition System on Google Glass for Assisting Social Interactions

They present a face recognition on Google Glass that is there to help with face detection, eye localization, face recognition and comes with a user interface for information display. Multiple methods are used to see which works better for localizing the eyes and different lighting conditions.

Article 4

Wearables Security and Privacy

Wearable devices are currently in many different applications in health care. Since these applications are relatively new there is little experience with security and privacy. It ends with the conclusion that before these wearables are for open market the security and privacy issues should be worked out.

Article 5

Facotrs Affecting Caregivers' Acceptance of the use of wearable devices by patiens with dementia: an extension of the unified theory of acceptance and use of technology model

Using a structered online questionnaire it was researched if caregivers will expect the use of wearables for patients with dementia. They constructed a model to see which factors had an influence. The conclusion is that social influence, effeort expectancy, facilitating conditions and behavioral intention all had a great impact on this model. But it was also seen that the technology anxiety and that people tend to have a resistance to chance were both not a predictor for the model.


MoSCoW Analysis

MUST

- have a camera

- have a microphone

- be able to be connected to an audio device (ear, headphones, connected by bluetooth, hearingaid)

- maintain a database with Images ('faces') and Audio fragments ('names') and  (‘relation’)

- operations on database: add

- have a user button1 pressed for a short amount of time to recognize faces

- have a user button1 pressed for a long amount of time to record audio fragment and take a picture to add to the database (photo + audio)

- have a button2 to switch on/off

- be easy to use


SHOULD

- be a closed system to protect data

- have a rechargeable battery and a device to charge

- have an indicator for low battery

- a housing to contain the hardware (easy to wear for instance around the neck)


COULD

- have a possibility of changing the audio volume (button3)

- send audio output to hearing device

- delete people from the database


WONT

- have a display

- edit the database


WEEKLY REPORTS?

W1
Matthieu Group discussion (2), Study papers (3) 5
Merel
Sam
Fee
Wilco
Dave


Progress Matthieu 6 maart

Ik heb met pyaudio library twee functions gemaakt:

1) Afspelen ingesproken audio fragment na korte spacebar press

2) Opnemen na langere (> halve seconde) op spacebar press

Opnemen maakt een wav file aan ('test.wav'). Ditzelfde fragment wordt afgespeeld.

Het is erg lastig om inspreken tijdens button press te realiseren, maar uiteindelijk zal dat wel lukken.

De uitdaging ligt er nu in om de boel op raspberry te krijgen en te koppelen met een oorttje. Ik heb raspberry besteld. Die krijg ik vandaag wss binnen.

12 March

Test environment complete: microphone, camera and ears on raspberry.

We got two working environments for audio and video part. This monday we will try to connect the two and make ik run on Raspberry.

WhatsApp Image 2023-03-12 at 13.21.01.jpg









Progress Sam

I worked on the recognizing face part of the software. Currently all the needed libraries are imported. I use the OpenCv and the facial_recognition libraries. With those libraries and LBPHFaceRecognizer provided from the OpenCv the facial recognition works. The current state is that the person needs to add pictures in a folder with a name and in that way the software will recognize the face. For now this is helpfull, but this needs to change from adding by the means of a folder, to adding by holding the button. And for the name instead of naming the folder it should use an audio part that the user itself gives. The following image can be obtained while using the webcam. (can someone help with the placement because this looks badd)

Face recognition.png






As you can see when a face is detected it will but a rectangle around that person. If the person is already added manually to the database it will present the name, in this case "Sam Lancee". When the person is not detected the label will say "unkown person".

Progress Matthieu & Sam 20-3-23

We focused on working together to implement both codes. It can now record an audio fragment and take a picture of the person that is in front of the camera by pressing the button for a couple of seconds and then it put them both in the same folder. After this the user can immediately press the buton once to detect the face and once the face is detected it will play the audio corresponding to that person.

Progress Matthieu 19-3

Got the system working on laptop. Flow between different modes or repetition of same mode is not fine yet!

While key pressed an audio is recorded (after a delay of 0.5 seconds)

If the key is released after the delay period (ie short press) the recognition is started. The systems recognizes 50-60% of the faces.

PROBLEM FOR NOW: implementation on raspberry. Since there is no keyboard on rasperry, i propose to simulate a key press by a rising GPIO and a key release by falling GPIO (button press)


_________________________________________________________________________

USER PART

INTRO

Dementia is a neurodegenerative disorder that affects several cognitive functions, such as memory, thinking, comprehension and language1 because of damage to brain cells. Dementia is a general term that covers different medical conditions, such as Alzheimer's disease2. Often, dementia affects people of higher age, however, dementia is not a direct consequence of age. Dementia is a progressive disease; many symptoms do get worse over time.  

There are different stages of the progress of dementia. The main stages are the early, middle and late stages, also called mild, moderate and severe 3. Often, the first stage of dementia is overlooked because the symptoms, such as losing track of the time and forgetfulness, are not seen as specific to dementia. During this stage, most people can live independently. The most well-known symptom during this stage is memory problems. During the second stage, patients may need more personal care because they experience changes in their behavior and might have difficulties while being at home. Problems with their memory become worse, they may for example confuse close ones with strangers. This may result in people struggling with their emotions because they might get upset about these changes.4 In the last stage of dementia, most aspects of a person's life will be affected. Difficulties with memory, mood swings, physical problems and changes in behavior will arise.  

Different technologies are invented to provide extra support in daily life for people who suffer from dementia. For example, different devices for helping with memory problems and socialising are already invented. A lot of people that do have dementia feel like these kinds of technologies are helpful for them and they are willing to use them as early as possible if it will help them.  

Because people with dementia often have difficulties with recognising faces and remembering the names that go with it, which may be very upsetting and frustrating, it might be helpful to invent a technology that supports the people with remembering faces of their close ones.  


Progress Dave

Introduction

This report will give a detailed user guide about the invention that will help people with memory loss remember names with the help from a wearable facial recognition scanner. The report will describe in detail why the device is needed in our society, for who the device is, what the exact goal of the device is and how the device works in general.


The user needs/requirements for the device

Nearly everyone has probably experienced the inconvenience of forgetting someone’s name, for most people this is not a severe problem as it does not happen distressingly often, but for over 155 million people in the world who suffer from some kind of form of memory loss [1] this is a serious problem as it can be embarrassing, distressing and often unhandy in formal settings. It could also lead to people being unnecessarily insulted if someone keeps on forgetting their name.

An example of this problem and a perceived solution in society is given now; Elly, a 73 year old woman, living in a retirement home was starting to suffer from dementia, to make things worse she knew it was happening. Every week her grandchildren went to visit her and she started to forget their names, Elly was terribly sad about it and thought to herself: “If only there was a device which would help me remember their names when I see them”

It can clearly be derived from the anecdote stated above that there is a serious need for a device which can help people remember names of loved ones. We will now state the needs/requirements that the device will have to have in a MoSCoW analysis stated below:

MUST

-  Have a camera

-  Be able to be connected to an audio device (ear, headphones, connected by Bluetooth, hearing aid)

-  Maintain a database with Images ('faces') and Audio fragments ('names') [maximum number of entries?]

-  Operations on database: add / delete (not edit)

- Have a user button1 to make an image of the person to be recognized, which triggers the recognition and plays the corresponding audio fragment on the ear

- Have a button2 to switch on/off

- Have a button3 to add an entry to the database (photo + audio)

- Be easy to use

SHOULD

-  Be closed system to protect data

-  Have a rechargeable battery and a device to charge

-  Have an indicator for low battery

-  A housing to contain the hardware (easy to wear for instance around the neck)


COULD

- Have a possibility of changing the audio volume (button4)

- Send audio output to hearing device

- Be waterproof

WON’T

- Have a display

- Be self-sustainable


The user group defined

The user group for the name remembering aid is huge, as stated above over 155 million people suffer from some kind of memory loss which makes it harder to remember names [1]. Some research even points out that nearly 9 out of 10 people sometimes have difficulties remembering the name of someone they met recently, that accounts to nearly 7.2 billion people! The device will be priced at a low price making it accessible for pretty much anyone who feels that forgetting people’s names is a problem they feel needs resolving. The device will be made as small and convenient as possible, so that it will be very user friendly to as many people as possible.

The target user group of the device will now be elaborated on further. When looking at the population of people with memory loss deficiencies the biggest deficiency by far is dementia, with over 55 million people suffering from this disease globally [3]. This target user group was chosen because this is a huge amount of people and a lot of research has been done on dementia, thus making it easier to get an image on how this disease effects its victims and how a device could best help these people.

About 75% of people with dementia are 70 years or older, so the majority of the target user group will be of elderly age, this will be taken into account when designing the device as it will have to be simple in function and also be very clear. Elderly people especially attach a lot of worth to being able to remember their loved ones, making them the best target user group which will benefit the most from the facial recognition device.

The device will be usable in all types of environments, but hospitals and care homes (where people with dementia often reside) will be the prime environment for the device. Because of this the device will be comfortable and easily adjustable, providing extra ease for our selected target user group.


Stigmatization of the device

When people buy a device they take in all aspects of it; the price, the aesthetics, the utility, but also how others would view them with the device. In this part of the report we are going to focus on the latter, analyzing how others think they would be perceived with the device.

There are a lot of devices on the market that help with disabilities that people face in their day to day lives, one of the most famous ones for example is the hearing aid. The first electrical hearing aid was invented in 1898 and has to this day been a popular wearable for people with hearing loss [5]. Research on the stigmatization of hearing aids has been done through multiple studies in the last decades, a recent study done two years ago found that 31% of people wearing hearing aids said that the device made them look visibly disabled [6], 28% said that it made people treat them differently and 34% said they felt embarrassed to wear them in public.

It is expected that our device will be sensitive to them same kind of stigmatic issues and that it might present a barrier for people buying them. To tackle this problem the name remembering aid will be made as discrete as possible so that the chance of it being noticed in public will be narrowed down. Making the device discrete will be a challenge but will be accomplished by the list of things beneath:


-       Make the device as small as possible.


-       Give the device a camouflage colour, which suits its environment.


-       Minimize the audio output so that only the user can hear the device clearly, but people around the user can barely hear it.


-       Make the design of the device in a form so that it is easy to put in pockets.


Research also shows that the attitude to the stigma around hearing aids is becoming better [7], more people are becoming familiar with it and are understanding it better which is leading to a higher acceptance rate in users. Users are also weighing the medical benefits up higher against the stigmatization negatives, resulting in more people choosing to wear hearing aid devices in public [7]. It is expected that the name remembering aid will follow the same positive trend as the one made by the hearing aids because the stigma around the name remembering device will be around the same as the one around hearing aids.


Privacy concerns

Privacy is a hot topic in modern day society, rightly so as we are constantly surrounded with devices that track and store information about us. This has caused a big impact on the market of electronics, especially ones with cameras and microphones, as people are starting to let the way that electronics handle their privacy influence their buying decisions [8]. It is thus of imperative importance that when releasing an electronic device, which uses cameras and microphones to scan faces and record voices, that is completely privacy safe and that the data observed by the device can not get in the wrong hands.

There are also multiple laws worldwide that restrict and prohibit certain capabilities of certain devices with cameras and microphones, and example is the ECPA law in the United States of America which was instated in 1968 [9].

The biggest concerns about privacy relating to electronic devices with facial recognition are stated below [10]:

-       Improper data storage


-       Misuse of data


-       Infringement on individual privacy


-       Infringement on freedom of speech and association


-       Lack of transparency


-       It might become normalized to abuse data privacy


-       New advances could make data abuse easy


The solution that our device brings to these concerns is to make the device a closed environment, this means that all data is stored internally in the device and that it can not be extracted (only physically). This means that the device its data with a common cloud based data base, thus making it impossible to extract information from the device without physically having access to it. The device will also be made to not unintentionally monitor its environment passively, without the user knowing, this gives the user control on when it wants the device to interact with its surroundings.

The risk of the user abusing the privacy of the surroundings he/she encounters does remain as the user will have psychical access to the device, this could be made harder by encrypting the data collected on the device but it will never be foolproof. It must though be stated though that this abuse could be done by any phone in this modern day and age, thus not making the device extraordinarily dangerous when compared to the standard smartphone, and probably much less dangerous as the name remembering aid is not linked with anything externally opposed to modern day smartphones.


The goal of the device

The main goal of the name remembering aid is simple, help the user remember the name of a person they are meeting in a discreet and quick manner without interrupting the social encounter, this will give the user a relaxed and reassuring feeling which will hopefully improve the ease of their day to day life.


How the device works

How the device works in general will now be explained. The making of a name remembering aid is not simple, the device needs to be small, discreet, accurate, safe and effective all at the same time to be able to serve its goal. The way how the device works is as followed: when you come into contact with someone the device will detect the faces of the people in your proximity, it will also listen to the people who you come into contact with and analyze the their speech searching for key phrases in which the name of the person is disclosed, the device then links the face to the assigned name and then stores this data internally (for privacy reasons). When the user meets the people again the device will recognize the faces and inform the user of the name linked to that face, if the face is new it will also inform the user so that he or she knows that they have never met this person in real life or that they did not state their name.


[1] Alzheimer's Facts and Figures Report | Alzheimer's Association

[2] Is Forgetting Words and Names a Sign of Dementia? - Health News Hub

[3] Dementia (who.int)

[4] What Are the Signs of Alzheimer's Disease? | National Institute on Aging (nih.gov)

[5] The History of Hearing Aids | Hearing Systems | Blog (hearingsystemsinc.com)

[6] Why Is There Stigma Around Wearing Hearing Aids? | Evergreen Speech and Hearing Clinic | Blog (everhear.com)

[7] The Stigma Surrounding Hearing Aids is Changing (hearingassociatesmc.com)

[8] Privacy policies influence consumer behavior online, new study finds | Press Releases | News | Newsroom | DePaul University, Chicago

[9] Electronic Communications Privacy Act of 1986 (ECPA) | Bureau of Justice Assistance (ojp.gov)

[10] 7 Biggest Privacy Concerns Around Facial Recognition Technology| LibertiesEU | liberties.eu


Device description

The product hangs around the users neck. Because of user-friendliness, the device has a limited number of buttons, only two buttons. The first button is the on/off button of the device. The second button has two functions. One is to recognize faces and the second one is to add people to the database. While someone continues to hold down the second button they can record audio, they can say the name and the relation of the person to the user. After releasing the button the device takes a picture and the audio recording and picture are added to the database. The device will train on the newly added picture so it can recognize the person added. Then when the user presses the second button for a short amount of time the device will start to recognize the face of the person in front of the camera. When the device recognizes the person in front of the camera, the audio fragment saying the name and relation of the person recognized, is played on the audio device.


Persona:

Persona 1








Differences between our product and other products

iCare Interaction Assistant for individuals who are visually impaired

The iCare Interaction Assistant is an device for helping people who are visually impaired during social interactions. In the paper ‘A Wearable Face Recognition System for Individuals with Visual Impairments’ (Krishna, S. 2005), it is said that face recognition is the initial step towards building a comprehensive social interaction assistant that will identify and interpret facial expressions, emotions and gestures. The device is part of the iCARE Interaction Assistant project that started with focus on supporting initial encounters, allowing a user to initiate social interactions. The goal is facilitating the subsequent verbal interaction by recognizing and interpreting non-verbal communication, including eye contact, facial expressions, emotions, and gestures.

The iCare Interaction Assistant had a different user group: people who are visually impaired. The device is more discrete, because the camera is in the bridge of a pair of glasses. The ultimate goal of this device is to help with non-verbal communication. Our product only helps with remembering names and relations.


Google Glass

In the paper, ‘A Wearable Face Recognition System on Google Glass for Assisting Social Interactions’ (Mendal, 2014) a wearable Face Recognition system on Google Glass for assisting people in social interactions is proposed. The proposed system works in two modes of operations: local (standalone) and client-server Bluetooth (with a mobile phone) architectures. It is a system that includes multiple face and eye detections and regularized subspace-based methods for training and testing individuals in an unconstrained environment.

In comparison to our product the proposed system is meant to aid all people in social interaction by recognizing persons and providing personal information. Our product focusses on people with dementia on recognizing persons and only providing names and relations.

It is useful to know that Google Glass is no longer available for consumers, but companies can still purchase Google Glass. Our proposed product should be available to consumers.


OrCam MyEye for blind and visually impaired people

OrCam MyEye is a voice activated device that attaches to glasses. The device can read text from a book, smartphone or other surfaces. It also recognizes faces. OrCam MyEye can help someone shop on their own, or work more efficiently. In this way the device can help blind or visually impaired people live a more independent life.

This product is meant for people that are blind or visually impaired not people with dementia. The size of the device is a as big as a finger and clicks to a pair of glasses. Our product is more visible because it is not attached to something else and is worn around the neck, and is slightly bigger. This device can do a lot more like reading text it also like our product recognizes faces and gives the associated names. The OrCam MyEye 2.0 price is regularly $4,500. This is very expensive for most people. We think our product can be made for a lot cheaper.


System specifications

Raspberry Pi 3 model A+

ArduCam 5 Megapixels 1080p sensor OV5647 mini camera video module. Angle of View: 54 x 41 degrees

USB Mini Multimedia Recording Voice Microphone

Minijack ears connected to Raspberry

Power source


Software

The python script uses the pyaudio library for audio recording. Face recognition is handled by opencv library.

Main functions in the script are:

  • def train() # training the dataset with newly taken images and audio fragment
  • def faces(dir_name=’’, mode=0) # Recognizes a face or store a face in dir_name # mode 0: Recognize a face and store image.png in parameter dir_name # mode 2: Take image of a face
  • def playAudioInDir(dirname) # Plays the file ‘audio_fragment.wav’ in folder dirname
  • def recordAudio(): # Records an audiofragment and stores it in dirname
  • def takePicture(dirname) # Runs faces in default mode 0
  • def startNewEntry() # starts recordAudio


Testing

Test faces


During testing we inserted images of test subjects into the system.

When showing the same images 5/24 were recognized.

When showing other images of the same persons, none of the persons was identified correctly.


Hardware and CAD

- Raspberry pi 3B+

- Picam V1.3 (5 MP It is to be determined still if the resolution is sufficient)

- Charge controller with battery board is ordered will arrive some time next week

Pictures CAD

Holding case raspberry pi 3b+
Screenshot 2023-02-16 165016.png



Updated CAD
Updated CAD


Pictures Test print

Test print 1.1.jpg
Test print 1.2.jpg


Test 2.1.jpg
Test 2.2.jpg


WRITING PART

INTRO

Dementia is a neurodegenerative disorder that affects several cognitive functions, such as memory, thinking, comprehension and language1 because of damage to brain cells. Dementia is a general term that covers different medical conditions, such as Alzheimer's disease2. Often, dementia affects people of higher age, however, dementia is not a direct consequence of age. Dementia is a progressive disease; many symptoms do get worse over time.

There are different stages of the progress of dementia. The main stages are the early, middle and late stages, also called mild, moderate and severe 3. Often, the first stage of dementia is overlooked because the symptoms, such as losing track of the time and forgetfulness, are not seen as specific to dementia. During this stage, most people can live independently. The most well-known symptom during this stage is memory problems. During the second stage, patients may need more personal care because they experience changes in their behavior and might have difficulties while being at home. Problems with their memory become worse, they may for example confuse close ones with strangers. This may result in people struggling with their emotions because they might get upset about these changes.4 In the last stage of dementia, most aspects of a person's life will be affected. Difficulties with memory, mood swings, physical problems and changes in behavior will arise.

Different technologies are invented to provide extra support in daily life for people who suffer from dementia. For example, different devices for helping with memory problems and socialising are already invented. A lot of people that do have dementia feel like these kinds of technologies are helpful for them and they are willing to use them as early as possible if it will help them.  

Because people with dementia often have difficulties with recognising faces and remembering the names that go with it, also called prosopagnosia, it might be helpful to invent a technology that supports the people with remembering faces of their close ones because it may be very upsetting and frustrating for both the patient and the close ones.

Therefore, in this report, it is explained how a facial recognition device could be an useful tool for people suffering from prosopagnosia.


DEFINING USERS

The device that is designed during this project, is designed for people suffering from dementia. It is a device that can hang around the neck of the user. The device includes a camera that detects faces that are across However, because of the bread understanding of the term dementia, it is important to define the users for the designed device more specifically. It is not possible for every patient to use the device. People that are in a later stage of dementia may have difficulties with understanding how to use the device. Talking more about the stages of dementia, it is possible to say that people who suffer from stage 3 of dementia are not capable enough to use the device. As written before, people with dementia in this stage, are very likely to experience many physical and behavior problems. Examples of these problems are losing the ability to control their muscles, needing help with daily activities and having increasing difficulties with communicating (https://dailycaring.com/3-stages-of-dementia-what-to-expect/).

The device consists of two different buttons. One button has to be used to save new faces including their names, and the other button is used to recognize the face opposite of the device and tells the user who they are facing. Therefore, in order to use the device, it is important that the user is capable of distinguishing the meaning of the two different buttons and of pressing the buttons long enough. Because people with stage 3 dementia are experiencing significant memory loss and thus different physical problems, these people are not included in the user group for the development of this device.

Also, people with stage 1 of dementia often do not have the symptoms yet for which the device is designed. The symptoms are, as mentioned before, sometimes even overlooked and problems with recognizing faces do not occur yet most of the time. Therefore, also this stage is excluded from our user group. This means that, for the people that suffer from dementia, only the ones in stage 2 are seen as the user group.


USER REQUIREMENTS

In order to design the best possible device, it is helpful to think about several requirements the device should have to make it the easiest for the user to use the device. Using the MoSCoW-method, different musts, shoulds, coulds and won’ts were thought of (moet deze zin veranderen maar weet nog niet zo goed hoe) (+ als bron erbij onze final Moscow uitschrijven in tabel/afbeelding). The device should have a camera in order to detect faces and these faces and the corresponding names should be saved in a database. More will be said about this later in the report.

Looking at the outside of the device, it is very important that the device is user friendly and easy enough to understand. Therefore, it is not useful if there are too many buttons. It is necessary that the device can be controlled and the amount of buttons is limited. This way, it is clear for the user which button has which function and how these buttons should be used.

Another requirement the device should have in order to simplify the use of the device is including a battery that lasts long enough. It would be very inconvenient for the user if the life of the battery is so short, that the device is not able to help the user for multiple hours.  


USER NEEDS

Different experience stories show that technology does have the potency to support people with dementia in their daily life. This can be done in different ways, such as providing safety but also keeping the user company (https://www.alzheimers.net/9-22-14-technology-for-dementia).

An woman that does have dementia, faces the difficulty that she starts forgetting the names of the people that are standing in front of her (https://www.cbc.ca/documentaries/the-nature-of-things/this-woman-can-t-recognize-her-own-daughter-s-face-but-technology-is-helping-her-1.6312174). This makes her insecure, however, a personalized app that is able to recognize people, helps her. As the woman says, it gives her certainty because she is able to have a conversation with someone without wondering what the name is of the one she is talking to.

This shows that technology can give support to people with prosopagnosia, but there should be a few requirements to fulfill the user’s needs. It is very important that the device supports the user by giving the user certainty. The device should be a tool that takes away the insecurity of the user for being unable to remember names and thus should be reliable. If the device is not reliable enough, and may for example even say wrong names, the insecurity of the user may even grow. If the device is reliable enough, it allows the user to be more independent and thus gives autonomy.

Concluding, it is very important that the device that will be developed is reliable. It should not make big mistakes such as telling the wrong names. If it is reliable enough, this will result in more certainty and autonomy from the user.


Stigmatization of the device

When people buy a device they take in all aspects of it; the price, the aesthetics, the utility, but also how others would view them with the device. In this part of the report we are going to focus on the latter, analyzing how others think they would be perceived with the device.

There are a lot of devices on the market that help with disabilities that people face in their day to day lives, one of the most famous ones for example is the hearing aid. The first electrical hearing aid was invented in 1898 and has to this day been a popular wearable for people with hearing loss [5]. Research on the stigmatization of hearing aids has been done through multiple studies in the last decades, a recent study done two years ago found that 31% of people wearing hearing aids said that the device made them look visibly disabled [6], 28% said that it made people treat them differently and 34% said they felt embarrassed to wear them in public.

It is expected that our device will be sensitive to them same kind of stigmatic issues and that it might present a barrier for people buying them. To tackle this problem the name remembering aid will be made as discrete as possible so that the chance of it being noticed in public will be narrowed down. Making the device discrete will be a challenge but will be accomplished by the list of things beneath:


-       Make the device as small as possible.


-       Give the device a camouflage colour, which suits its environment.


-       Minimize the audio output so that only the user can hear the device clearly, but people around the user can barely hear it.


-       Make the design of the device in a form so that it is easy to put in pockets.


Research also shows that the attitude to the stigma around hearing aids is becoming better [7], more people are becoming familiar with it and are understanding it better which is leading to a higher acceptance rate in users. Users are also weighing the medical benefits up higher against the stigmatization negatives, resulting in more people choosing to wear hearing aid devices in public [7]. It is expected that the name remembering aid will follow the same positive trend as the one made by the hearing aids because the stigma around the name remembering device will be around the same as the one around hearing aids.


Privacy concerns

Privacy is a hot topic in modern day society, rightly so as we are constantly surrounded with devices that track and store information about us. This has caused a big impact on the market of electronics, especially ones with cameras and microphones, as people are starting to let the way that electronics handle their privacy influence their buying decisions [8]. It is thus of imperative importance that when releasing an electronic device, which uses cameras and microphones to scan faces and record voices, that is completely privacy safe and that the data observed by the device can not get in the wrong hands.

There are also multiple laws worldwide that restrict and prohibit certain capabilities of certain devices with cameras and microphones, and example is the ECPA law in the United States of America which was instated in 1968 [9].

The biggest concerns about privacy relating to electronic devices with facial recognition are stated below [10]:

-       Improper data storage


-       Misuse of data


-       Infringement on individual privacy


-       Infringement on freedom of speech and association


-       Lack of transparency


-       It might become normalized to abuse data privacy


-       New advances could make data abuse easy


The solution that our device brings to these concerns is to make the device a closed environment, this means that all data is stored internally in the device and that it can not be extracted (only physically). This means that the device its data with a common cloud based data base, thus making it impossible to extract information from the device without physically having access to it. The device will also be made to not unintentionally monitor its environment passively, without the user knowing, this gives the user control on when it wants the device to interact with its surroundings.

The risk of the user abusing the privacy of the surroundings he/she encounters does remain as the user will have psychical access to the device, this could be made harder by encrypting the data collected on the device but it will never be foolproof. It must though be stated though that this abuse could be done by any phone in this modern day and age, thus not making the device extraordinarily dangerous when compared to the standard smartphone, and probably much less dangerous as the name remembering aid is not linked with anything externally opposed to modern day smartphones.