View source for Pantic, M. et al. (2007). Human Computing and Machine Understanding of Human Behavior: A Survey.

How can a robot understand human social behavior? M. Pantic et al. aim to somewhat quantize this question in their paper, by explaining all the important factors that pertain to understanding human social cues. There are three main channels through which humans socially communicate: tactile, visual and auditory.


These three channels help the robot perceive three important areas to understand social behavior: face, body, and vocal nonlinguistic signals (like laughs, shouts and winks). The robot needs to pay close attention to these three things if it wishes to understand the human on a social level. Furthermore, the robot also needs to sense the context, because this is a big part of social interaction. The robot must identify six aspects of the environment for optimal performance. These are also referred to as the W5+:

1. Who? (Who is the user?)

2. Where? (Where is the user?)

3. What? (What is the current task of the user?)

4. How? (How is information passed on by the user?)

5. When? (What is the timing of displayed behavioral signals with respect to changes in the environment?)

6. Why? (Why does the user display observed cues?)


In the end, if robots can sense human aspects, like body, face and nonlinguistic vocal cues, and if it can answer the W5+, it should be able to read and understand human social behavior quite well. To make it a little more concrete, the two goals to be attained in this field are: understanding human affect (1), and understanding human social signalling (2).