Clairvoyance

To return to the home page click here.

To go to Coaching questions page click here.

Introduction and Description

Humans have been taking care of their hair since the Palaeolithic age, and nowadays hairstyles have become a staple in many different cultures and come in a variety of styles. In the western world, trends are usually established when celebrities appear in public with a new haircut, this often leads to a large number of people wanting a similar haircut. People rush to their hairdressers in demand for the exact same haircut not worried about the fact that it might look differently on them than it did on that celebrity. This can be due to various reasons like the difference in hair thickness, head shape or even lack of specific high-quality products and care needed to produce and maintain the haircuts. There is no real way to visualize the haircut on oneself apart from photoshopping images or using apps that give a poor representation. This is where Clairvoyance comes in.

There are multiple scenarios in which Clairvoyance can excel. Here are two of them.

Scenario 1

You walk into the hairdresser's with a haircut in mind, you explain it to your hairdresser and they seem to understand it. They start cutting and styling your hair only for you to realise at the end that this is not what you wanted. The haircut you received is not what you expected the haircut to look like on you, and that you've actually prefered another haircut. There is no way, however, to roll back to before you got the haircut and you are stuck with an unsatisfactory hair do. With this modern product, you will now be able to avoid all these unnecessary and annoying issues.

Scenario 2

You're a hairdresser and a customer walks into your store with an idea in mind. They present you that idea to the best of their ability, and you start cutting, styling and/or colouring their hair. When you present them with the final result they seem unsatisfied. This can be due to numerous reasons, you could simply have made a mistake, but most likely the haircut or style they requested is not what they expected it to be.

Description

The Clairvoyance preview imaging technology would allow for an easy bridge between a customer's expectation and a hairdresser's understanding! This can help them make a final decision on a haircut as well as concretise the idea customers have in mind. The product would allow for hairstyle and hair colour to be displayed and be easily customizable. No more miscommunications or disappointed customers.

Existing Work

Research

Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression

State of the art AI technique for reconstructing facial meshes from a single picture. Does so by using a novel neural network technique called volumetric CNNs. For Clairvoyance this is a large piece of the puzzle since it would let us create realistic renderings of person's face with a different hairstyle. It is not the whole puzzle though; see the section drawbacks below. An online interactive demo of this technique is available at http://cvl-demos.cs.nott.ac.uk/vrn/

Head Reconstruction from Internet Photos

Reconstructs a whole head from hundredths of images. Results of the paper do not appear good enough for production.

3D Face Hallucination from a Single Depth Frame

Reconstructs a whole head from a single depth camera image. Impressive results.

Existing products

(Maybe explain why our application is better than the applications already available. They seem more like toys, while our idea might actually be useful for both customers and hairdressers.)

3D Reconstruction Products

facegen

Creates a 3D model of a person's head from a single or multiple images. However, final results are not very personalized but instead are constructed from an existing database of the heads of many different people. The results seem more like avatars than realistic 3D models.

itseez3d

Reconstructs any object in 3D by recording 360 degrees around an object with an iPad. Requires the Structure Sensor third-party peripheral.

Challenges

See the existing work section first. We have found very little mention in literature or otherwise of automatically generating a 3D model of a person's whole head from a few images. Existing research seems not yet ready for production or requires a calibrated setup where the background is eliminated, lighting is constant and motion of the camera is controlled. There also exists work where relatively cheap depth cameras are used that provide impressive results, depth cameras are however not yet available on most mobile devices. However, there is promising work in reconstructing the face part of a head. It is possible to generalize the rest of a person's head. That is, "glue" the face part on top of an average head model. The drawback of this is that we lose important details (in the case of applying hairstyles) such as the shape of a person's ears.

User Survey

Two surveys were created in order to receive feedback about our product. One was created for regular people and the other for hairdressers because we expect both groups to be able to deliver valuable information, but also have differences of opinions. Professional application of this device may require different features and expectation than that of customers. The surveys were made using the SurveyMonkey.com site. This site allows for an easy distribution of the survey as well as a quick collection and analysis of the data.

The survey starts off with a normal life scenario without the product so people can relate to the problematic situation. This differs between hairdressers and their customers. The product is then described and explanations are given to show where the product's strengths are and how it can improve the user's life. After this explanation, a list of questions is given from which we aim to extract as much information about the feelings and expectations for the product.

The survey intended for the professional use of the product only differs in its description, in order help them relate more to the problem at hand.

Question 1

The first question helps us get a feel for the initial opinion of the user towards this product. This is very useful because it gives us a good idea of how people feel about a product after a first description and thus how likely it is that the users want the product. Five answers are then available which range from very negative to very positive to allow the user to express their emotions towards the product.

Question 2

Quality is an important factor for product development since it will influence the price the product will be retailed at. More importantly, it also gives us an idea of how reliable and dependent our product is.

Question 3

This question asks how innovative users think the product is. This gives us insight on whether our product is perceived as state of the art and how much it will impact the market of the targeted industry, in this case, the hair industry.

Question 4

Question 4 asks how necessary the product seems to the users. Higher necessity will lead to a higher demand for the product. We expect beauty salon's and hairdressers to have a particularly high need for the product since it can help them distinguish their establishment from those without this product.

Question 5

This gives us an idea of how many people would buy this product and how enthusiastic users are for it. Furthermore, it complements the previous question on the necessity of the product.

Question 6

For this question, the survey changes between the user version and the professional version. In the professional version, we aim to see to what extent they would use this product over whatever method they are currently using. The user version asks how it will affect their habits if they have this product or the hairdresser does, using this information we can see how beneficial and profitable the product is for the professionals.

User version

Professional version

Question 7

With this question, we can find out which elements people find most important for our product. This can help us focus our attention towards specific attributes to improve our user's satisfaction as well as solve as many of their issues as possible. It's important to remember however that all these aspects, even if they are not very popular, will be important for the product. And that users may find some parts crucial and only realise it when they aren't optimised.

Question 8

This question gives an idea of the rate of diffusion of the product. In addition, it shows how enthusiastic the user is about the product and its innovations.

Question 9

Question 9 features an open question which allows the user to give opinions about the strong points of this product. It shows what they find most important in the product and how it can improve their lives.

Question 10

The final question in the questionnaire allows people to give feedback and suggest improvements to our technology. In this way, we can make sure to not miss out on valuable features that complement this technology as well be aware of the flaws in the product and its uses.

Laws and Ethics

Laws surrounding the issue

The idea of the mirror doesn’t violate any of the human rights since you voluntarily visit the hairdresser and can still decide to not have your hair cut after you dislike what you see. It is legal to put a price tag on the ‘’hair scanning’’ because it takes time and labour hours can legally be charged. The idea even really nicely complies with Article 27.

Everyone has the right freely to participate in the cultural life of the community, to enjoy the arts and to share in scientific advancement and its benefits.

As the concept of the mirror is very approachable for people from all classes, genders/cultures or religions, it is very easy to let everybody share in this scientific advancement and its benefits.

Privacy

Privacy is a fundamental human right recognized in the UN Declaration of Human Rights, the International Covenant on Civil and Political Rights and in many other international and regional treaties.

In the Netherlands, a business must follow the Dutch Data Protection Act (Wet bescherming persoonsgegevens, Wbp). This act outlines and emphasizes the amount of care needed when handling a customer's personal data. Without such acts, the data would be vulnerable to criminal activity such as theft. Transparency, with the user, about the usage of the data is crucial, they should be informed of where and how their data is being used. In some cases, the data must be reported to the Dutch Data Protection Authority, DPA (Autoriteit Persoonsgegevens). In addition, the customer should be made aware of your details such as your address and company names as well as making sure they are informed when you intend to share their data with other companies. It is furthermore, mandatory to include a privacy statement on your website if you have one.

A model that can be used when dealing with privacy factors in multimedia information sharing context is what we will refer to here as the Adams' privacy model. The aim of this model is to help multimedia communication systems to "determine which information users regard as private, from whom, and in which context". Rather than dealing with privacy as a black/white (or binary yes/no) variable, the model attempts to define "privacy boundaries" which if breached will reduce the effectiveness of the system being used. Furthermore, what defines these boundaries is, in fact, the privacy "perception" of the users. According to Adams' model, the User is anyone who has their data transmitted either directly or indirectly. Direct information may, for instance, be a person's consumption habits or medical records, while indirect information might be their image or voice. In fact, the user may not actually be actively using the system and may be totally unaware that their data is being transmitted. As shown in the figure below, Adams' model defines three main factors which interact with one another to create the users' overall perception of their privacy. These factors are defined as:

• Information Sensitivity: is the primary privacy factor which the other factors affect to determine the perceived sensitivity level. Information Sensitivity in a way defines the users' perceptions of the confidentiality of the information being transmitted. Also, users' judgment of the sensitivity levels of the information is not binary (private/not private), but multi-dimensional with varying degrees of sensitivity.

• Information Receiver: is the users' perception of the person who receives and/or manipulates their information. Although a range of factors would influence the users' assessment of the Information Receiver, the issue of trust seems to be the most important one in defining the users' perception of the Information Receiver.

• Information Usage: is the users' perception of how and for what purpose their information will be used at present as well as in the future. The potential importance that the users attribute to the perceived Information Usage is crucial to the users' estimation of privacy risk/benefit trade-offs.

A model that can be used when dealing with privacy factors in multimedia information sharing context is what we will refer to here as the Adams' privacy model. The aim of this model is to help multimedia communication systems to "determine which information users regard as private, from whom, and in which context". Rather than dealing with privacy as a black/white (or binary yes/no) variable, the model attempts to define "privacy boundaries" which if breached will reduce the effectiveness of the system being used. Furthermore, what defines these boundaries is, in fact, the privacy "perception" of the users. According to Adams´ model, the User is anyone who has their data transmitted either directly or indirectly. Direct information may, for instance, be a person's consumption habits or medical records, while indirect information might be their image or voice. In fact, the user may not actually be actively using the system and may be totally unaware that their data is being transmitted. As shown in the figure below, Adams´ model defines three main factors which interact with one another to create the users´ overall perception of their privacy. These factors are defined as:

• Information Sensitivity: is the primary privacy factor which the other factors affect to determine the perceived sensitivity level. Information Sensitivity in a way defines the users' perceptions of the confidentiality of the information being transmitted. Also, users' judgment of the sensitivity levels of the information is not binary (private/not private), but multi-dimensional with varying degrees of sensitivity.

• Information Receiver: is the users' perception of the person who receives and/or manipulates their information. Although a range of factors would influence the users' assessment of the Information Receiver, the issue of trust seems to be the most important one in defining the users´ perception of the Information Receiver.

• Information Usage: is the users' perception of how and for what purpose their information will be used at present as well as in the future. The potential importance that the users attribute to the perceived Information Usage is crucial to the users´ estimation of privacy risk/benefit trade-offs.

Requirements, Preferences and Constraints

Requirements

1. The application allows the user to take pictures and form a picture set.
2. A picture set consists of 4 pictures, capturing the front, back, left and right side of the user's head.
3. The application has to be able to create a 3D model of the user's head out of a picture set.
4. The application is able to extract the specific side of the user's head from each picture in the picture set.
5. The application has to be able to apply a 3D model of a haircut on the head of the 3D model of the person’s head.
6. The user should be able to manually choose and change the applied hairstyle.
7. The application must be able to show the results of applying a 3D model of a hairstyle on the head of the 3D model of the person’s head (3D view).
8. The user is able to pan the 3D view by swiping left and right; thereby allowing the user to see the whole of his generated head and applied hairstyle.

Preferences

1. The product should offer a well-received user experience (as determined by user tests).
2. The product should offer a tutorial in which the user is told how to use the product.
3. The application will initially contain at least 3 hairstyles per gender (male and female).
4. The application will allow changing the colour of the hairstyles.
5. The application will be able to download and use new hairstyles from a central server hosting our hairstyle files.
6. The hairdresser can pre-select which hairstyles he/she is able to cut/style.
7. The hairdresser can pre-select which hairstyles can actually be cut/styled for the current client.
8. The application will provide a realistic lighting solution for the 3D view.
9. The user is able to select different lighting.
10. The user is able to select light colour.
11. The user is able to select light position.
12. The user is able to select light intensity.

Constraints

1. The application must run in iOS 10 devices.
2. The application must run suitably (as determined by user tests) on a fifth-generation iPad or later iPads.
3. The application must run on Android 8 devices.
4. The application must run suitably (as determined by user tests) on a Samsung Galaxy Tab S3.
5. The application must run suitably (as determined by user tests) on an Asus ZenPad 3S 10.

Prototype

The prototype will consist of an Android app, showcasing our vision in terms of user experience and functionality. Its primary purpose is to gauge interest in the final product to potential buyers and users. (Pictures of the prototype will follow)

The prototype will have:

• User interface to take 4 pictures of the user.
• A 3D view showing a 3D mesh of the user's head.
• At least 2 different hairstyles. The exact number will depend on the time required to create (or download and use) a single hairstyle (TBD).
• User interface to select different hairstyles.
• The 3D view shows the selected hairstyle on top of the user's head.
• The 3D view is rotatable so that user can see his whole head with the selected hairstyle.

The prototype will not have:

• The ability to convert from picture to 3D mesh. Instead, we will pre-generate a 3D mesh.
• The ability to dynamically add the hair to the 3D mesh. Instead, we will pre-generate this.
• A realistic lighting solution. Instead, we will use a simpler lightning solution.

Discussion

In this section, we will focus on the what we were not able to do during this project, as well as areas where increased research and experimentation is needed.

Technical features

Since what we have as our final product for this course is a prototype, we don't have a fully functioning product. This prototype doesn't provide some important technical features such as:

High-quality resolution of the hair

The current resolution of the hair is simply what Maya has to offer and while this is useful for a prototype, customers will request more accurate and realistic hair in order to improve the immersion of the application.

Hair type

Although trends and culture can cause hairstyles to be similar within communities, there are many hair types that need to be taken into account. From straight hair to afros, hair varies in curliness, in fact, many people choose to go from one hair type to another. The current application does not take this into account due to the number of hairstyles made in the limited amount of time. This is an important feature because people sometimes enjoy straightening or curling their hair for a change and therefore seeing the possible drastic result is crucial.

Hair length is another factor that should not be ignored, although people can't instantly grow their hair at the hairdresser, people let their hair grow different amounts based on preference and sometimes people decide to go for drastic changes.

The thickness and the amount of hair on are variables affected by genes, age, and even stress. Many hairstyles require a certain degree of these variables in order to hold or even just to look similar. This is obviously an essential part of the final product.

Accurate hair movement and physics when rotating the face

Since hair is made up of hundreds of thousands of thin hair strands, its individual as well general position can be changed by just a small breeze. Because we live actively and constantly move our heads and bodies around, our hair is subjected to this movement and moves depending on its length and the force issued onto it. In order to improve the realism of the application hair movement can be added such that people can visualize how their hair would move around and relate more easily to the displayed 3D image.

Hair reflectivity from various light sources

Although this is a minor adjustment, it might be interesting to add different settings in which the hair is illuminated by different light sources and light types.

Accurate real-time customizability such as

• Hair length customization
• Hair type (thickness or curls)
• Hair colour

Hair customizability is a crucial feature in this application because its the primary component that facilitates the interaction between the hairdresser and their customer. Furthermore, it allows for more creative freedom as well as quick visualization of the expected final product. Although our prototype already fulfils some of these features, it does so in a limited fashion and thus doesn't represent the full potential of these features.

App developments

Furthermore, some developments could be made to the app itself in regards to:

Server storage of data for personal use application

The prototype does not have a way to store personal information but this would be part of the final product. In this way, users will have a history of their haircuts that can be shared with any hairdresser and possibly any of their friends. This, however, requires backend services as well as a thorough understanding of privacy which has been expanded on in this wiki.

UI improvements to make the app look better

The current interface is simple and effective, but by no means is it optimal in terms of aesthetic and design. This can be achieved by hiring a good frontend designer or by relying on user feedback to improve it.

Ratings in terms of difficulty for the desired hairstyle to be realised

If the haircut you desire is quite complex to get right it might be more effective to go to a higher end hairdresser.

Social Network

Additionally, it would be interesting to create a social network for the users which allows them to

• Share haircuts they have created using the app
• Rate and comment on other haircuts
• Rate and comment on hairdressers
• Save haircuts to a favourites tab to see how it would look on them