Emotions with Gaze
Description: Humans express emotions via various modalities, such as with facial expressions, vocality, and gestural prompts. Surface forms of emotion expression have been researched and observed on social media platforms (via natural language processing techniques), through speech analysis (within the domain of digital phonetics and signal processing), facial cues (within the domain of computer vision), etc., where all domains of research are integrating aspects of cognitive science for their particular tasks.
The notion behind this project would be to find a relationship between gaze and affect expression, as well as speech, leveraging gaze information in order to predict emotion classes. The project would be to use two existing multimodal datasets in which annotations of facial landmarks, speech data, and emotion labels are present. Then the student would use a tool, such as Openface: A general-purpose face recognition library with mobile applications (Amos et al., 2016), for gaze estimation to investigate/analyze the temporal relationship between gaze, emotion, and speech. The student will explore joint representations/multimodal embeddings of speech, gaze, facial features, etc. The goal would be to use a machine learning approach — to build a classifier in order to predict various emotions using facial expressions, gaze information, and speech. Then they would test the model on the subset of videos unseen during training (perhaps also evaluate their model(s) with adversarial attacks via “fake emotional” videos, i.e movies, etc). The project is about exploring various human joint multimodal representations in human emotion expression — various modalities coupled with affect expression: gaze, facial expressions, and speech.
Supervisor: Ekta Sood
Distribution: 20% Literature, 30% Data Collection, 30% Implementation, 20% Data Analysis and Evaluation.
Requirements: Interest in affective computing, emotion analysis, and multimodal machine learning approaches, familiar with data processing and analysis/statistics, experience with machine learning and the following frameworks — Tensorflow/PyTorch/Keras.
Literature: Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. 2016. Openface: A general-purpose face recognition library with mobile applications. CMU School of Computer Science, Technical Report CMU-CS-16-118.