Work featured in 'Max Planck Forschung'

Our work was featured as a focus topic in the current “Max Planck Forschung” magazine (local link).

1. 

   MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation

   Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling

   IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 41(1), pp. 162-175, 2019.

   Abstract Links BibTeX Project

   Learning-based methods are believed to work well for unconstrained gaze estimation, i.e. gaze estimation from a monocular RGB camera without assumptions regarding user, environment, or camera. However, current gaze datasets were collected under laboratory conditions and methods were not evaluated across multiple datasets. Our work makes three contributions towards addressing these limitations. First, we present the MPIIGaze dataset, which contains 213,659 full face images and corresponding ground-truth gaze positions collected from 15 users during everyday laptop use over several months. An experience sampling approach ensured continuous gaze and head poses and realistic variation in eye appearance and illumination. To facilitate cross-dataset evaluations, 37,667 images were manually annotated with eye corners, mouth corners, and pupil centres. Second, we present an extensive evaluation of state-of-the-art gaze estimation methods on three current datasets, including MPIIGaze. We study key challenges including target gaze range, illumination conditions, and facial appearance variation. We show that image resolution and the use of both eyes affect gaze estimation performance, while head pose and pupil centre information are less informative. Finally, we propose GazeNet, the first deep appearance-based gaze estimation method. GazeNet improves on the state of the art by 22% (from a mean error of 13.9 degrees to 10.8 degrees) for the most challenging cross-dataset evaluation.

   doi: 10.1109/TPAMI.2017.2778103

   Paper: zhang19_pami.pdf

   @article{zhang19_pami, title = {MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation}, author = {Zhang, Xucong and Sugano, Yusuke and Fritz, Mario and Bulling, Andreas}, year = {2019}, journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)}, doi = {10.1109/TPAMI.2017.2778103}, pages = {162-175}, volume = {41}, number = {1} }

1. 

   Emergent Leadership Detection Across Datasets

   Philipp Müller, Andreas Bulling

   Proc. ACM International Conference on Multimodal Interaction (ICMI), pp. 274-278, 2019.

   Abstract Links BibTeX Project

   Automatic detection of emergent leaders in small groups from nonverbal behaviour is a growing research topic in social signal processing but existing methods were evaluated on single datasets – an unrealistic assumption for real-world applications in which systems are required to also work in settings unseen at training time. It therefore remains unclear whether current methods for emergent leadership detection generalise to similar but new settings and to which extent. To overcome this limitation, we are the first to study a cross-dataset evaluation setting for the emergent leadership detection task. We provide evaluations for within- and cross-dataset prediction using two current datasets (PAVIS and MPIIGroupInteraction), as well as an investigation on the robustness of commonly used feature channels and online prediction in the cross-dataset setting. Our evaluations show that using pose and eye contact based features, cross-dataset prediction is possible with an accuracy of 0.68, as such providing another important piece of the puzzle towards real-world emergent leadership detection.

   doi: 10.1145/3340555.3353721

   Paper: mueller19_icmi.pdf

   @inproceedings{mueller19_icmi, title = {Emergent Leadership Detection Across Datasets}, author = {M{\"{u}}ller, Philipp and Bulling, Andreas}, year = {2019}, pages = {274-278}, booktitle = {Proc. ACM International Conference on Multimodal Interaction (ICMI)}, doi = {10.1145/3340555.3353721} }
2. 

   PrivacEye: Privacy-Preserving Head-Mounted Eye Tracking Using Egocentric Scene Image and Eye Movement Features

   Julian Steil, Marion Koelle, Wilko Heuten, Susanne Boll, Andreas Bulling

   Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1–10, 2019.

   Abstract Links BibTeX Project Best video award

   Eyewear devices, such as augmented reality displays, increasingly integrate eye tracking but the first-person camera required to map a user’s gaze to the visual scene can pose a significant threat to user and bystander privacy. We present PrivacEye, a method to detect privacy-sensitive everyday situations and automatically enable and disable the eye tracker’s first-person camera using a mechanical shutter. To close the shutter in privacy-sensitive situations, the method uses a deep representation of the first-person video combined with rich features that encode users’ eye movements. To open the shutter without visual input, PrivacEye detects changes in users’ eye movements alone to gauge changes in the "privacy level" of the current situation. We evaluate our method on a first-person video dataset recorded in daily life situations of 17 participants, annotated by themselves for privacy sensitivity, and show that our method is effective in preserving privacy in this challenging setting.

   doi: 10.1145/3314111.3319913

   Paper: steil19_etra.pdf

   Supplementary Material: steil19_etra_sup.pdf

   Video: https://www.youtube.com/watch?v=Gy61255F8T8

   @inproceedings{steil19_etra, title = {PrivacEye: Privacy-Preserving Head-Mounted Eye Tracking Using Egocentric Scene Image and Eye Movement Features}, author = {Steil, Julian and Koelle, Marion and Heuten, Wilko and Boll, Susanne and Bulling, Andreas}, year = {2019}, booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)}, pages = {1--10}, doi = {10.1145/3314111.3319913}, video = {https://www.youtube.com/watch?v=Gy61255F8T8} }
3. 

   Privacy-Aware Eye Tracking Using Differential Privacy

   Julian Steil, Inken Hagestedt, Michael Xuelin Huang, Andreas Bulling

   Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1–9, 2019.

   Abstract Links BibTeX Project Best paper award

   With eye tracking being increasingly integrated into virtual and augmented reality (VR/AR) head-mounted displays, preserving users’ privacy is an ever more important, yet under-explored, topic in the eye tracking community. We report a large-scale online survey (N=124) on privacy aspects of eye tracking that provides the first comprehensive account of with whom, for which services, and to which extent users are willing to share their gaze data. Using these insights, we design a privacy-aware VR interface that uses differential privacy, which we evaluate on a new 20-participant dataset for two privacy sensitive tasks: We show that our method can prevent user re-identification and protect gender information while maintaining high performance for gaze-based document type classification. Our results highlight the privacy challenges particular to gaze data and demonstrate that differential privacy is a potential means to address them. Thus, this paper lays important foundations for future research on privacy-aware gaze interfaces.

   doi: 10.1145/3314111.3319915

   Paper: steil19_etra_2.pdf

   @inproceedings{steil19_etra_2, title = {Privacy-Aware Eye Tracking Using Differential Privacy}, author = {Steil, Julian and Hagestedt, Inken and Huang, Michael Xuelin and Bulling, Andreas}, year = {2019}, booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)}, doi = {10.1145/3314111.3319915}, pages = {1--9} }

1. 

   Eye movements during everyday behavior predict personality traits

   Sabrina Hoppe, Tobias Loetscher, Stephanie Morey, Andreas Bulling

   Frontiers in Human Neuroscience, 12, pp. 1–8, 2018.

   Abstract Links BibTeX Project

   Besides allowing us to perceive our surroundings, eye movements are also a window into our mind and a rich source of information on who we are, how we feel, and what we do. Here we show that eye movements during an everyday task predict aspects of our personality. We tracked eye movements of 42 participants while they ran an errand on a university campus and subsequently assessed their personality traits using well-established questionnaires. Using a state-of-the-art machine learning method and a rich set of features encoding different eye movement characteristics, we were able to reliably predict four of the Big Five personality traits (neuroticism, extraversion, agreeableness, conscientiousness) as well as perceptual curiosity only from eye movements. Further analysis revealed new relations between previously neglected eye movement characteristics and personality. Our findings demonstrate a considerable influence of personality on everyday eye movement control, thereby complementing earlier studies in laboratory settings. Improving automatic recognition and interpretation of human social signals is an important endeavor, enabling innovative design of human–computer systems capable of sensing spontaneous natural user behavior to facilitate efficient interaction and personalization.

   doi: 10.3389/fnhum.2018.00105

   Paper: hoppe18_fhns.pdf

   @article{hoppe18_fhns, title = {Eye movements during everyday behavior predict personality traits}, author = {Hoppe, Sabrina and Loetscher, Tobias and Morey, Stephanie and Bulling, Andreas}, doi = {10.3389/fnhum.2018.00105}, volume = {12}, pages = {1--8}, year = {2018}, journal = {Frontiers in Human Neuroscience} }

1. 

   Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors

   Julian Steil, Philipp Müller, Yusuke Sugano, Andreas Bulling

   Proc. ACM International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI), pp. 1–13, 2018.

   Abstract Links BibTeX Project Best paper award

   Visual attention is highly fragmented during mobile interactions but the erratic nature of attention shifts currently limits attentive user interfaces to adapt after the fact, i.e. after shifts have already happened. We instead study attention forecasting – the challenging task of predicting users’ gaze behavior (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users’ visual scene. We demonstrate that our method can forecast bidirectional attention shifts and whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.

   doi: 10.1145/3229434.3229439

   Paper: steil18_mobilehci.pdf

   @inproceedings{steil18_mobilehci, author = {Steil, Julian and M{\"{u}}ller, Philipp and Sugano, Yusuke and Bulling, Andreas}, title = {Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors}, booktitle = {Proc. ACM International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI)}, year = {2018}, doi = {10.1145/3229434.3229439}, pages = {1--13} }
2. 

   Robust Eye Contact Detection in Natural Multi-Person Interactions Using Gaze and Speaking Behaviour

   Philipp Müller, Michael Xuelin Huang, Xucong Zhang, Andreas Bulling

   Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1–10, 2018.

   Abstract Links BibTeX Project

   Eye contact is one of the most important non-verbal social cues and fundamental to human interactions. However, detecting eye contact without specialized eye tracking equipment poses significant challenges, particularly for multiple people in real-world settings. We present a novel method to robustly detect eye contact in natural three- and four-person interactions using off-the-shelf ambient cameras. Our method exploits that, during conversations, people tend to look at the person who is currently speaking. Harnessing the correlation between people’s gaze and speaking behaviour therefore allows our method to automatically acquire training data during deployment and adaptively train eye contact detectors for each target user. We empirically evaluate the performance of our method on a recent dataset of natural group interactions and demonstrate that it achieves a relative improvement over the state-of-the-art method of more than 60%, and also improves over a head pose based baseline.

   doi: 10.1145/3204493.3204549

   Paper: mueller18_etra.pdf

   @inproceedings{mueller18_etra, author = {M{\"{u}}ller, Philipp and Huang, Michael Xuelin and Zhang, Xucong and Bulling, Andreas}, title = {Robust Eye Contact Detection in Natural Multi-Person Interactions Using Gaze and Speaking Behaviour}, booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2018}, pages = {1--10}, doi = {10.1145/3204493.3204549} }
3. 

   Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behavior

   Philipp Müller, Michael Xuelin Huang, Andreas Bulling

   Proc. ACM International Conference on Intelligent User Interfaces (IUI), pp. 153-164, 2018.

   Abstract Links BibTeX Project

   Rapport, the close and harmonious relationship in which interaction partners are "in sync" with each other, was shown to result in smoother social interactions, improved collaboration, and improved interpersonal outcomes. In this work, we are first to investigate automatic prediction of low rapport during natural interactions within small groups. This task is challenging given that rapport only manifests in subtle non-verbal signals that are, in addition, subject to influences of group dynamics as well as inter-personal idiosyncrasies. We record videos of unscripted discussions of three to four people using a multi-view camera system and microphones. We analyse a rich set of non-verbal signals for rapport detection, namely facial expressions, hand motion, gaze, speaker turns, and speech prosody. Using facial features, we can detect low rapport with an average precision of 0.7 (chance level at 0.25), while incorporating prior knowledge of participants’ personalities can even achieve early prediction without a drop in performance. We further provide a detailed analysis of different feature sets and the amount of information contained in different temporal segments of the interactions.

   doi: 10.1145/3172944.3172969

   Paper: mueller18_iui.pdf

   @inproceedings{mueller18_iui, title = {Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behavior}, author = {M{\"{u}}ller, Philipp and Huang, Michael Xuelin and Bulling, Andreas}, year = {2018}, pages = {153-164}, booktitle = {Proc. ACM International Conference on Intelligent User Interfaces (IUI)}, doi = {10.1145/3172944.3172969} }

1. 

   InvisibleEye: Mobile Eye Tracking Using Multiple Low-Resolution Cameras and Learning-Based Gaze Estimation

   Marc Tonsen, Julian Steil, Yusuke Sugano, Andreas Bulling

   Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 1(3), pp. 1–21, 2017.

   Abstract Links BibTeX Project Distinguished paper award

   Analysis of everyday human gaze behaviour has significant potential for ubiquitous computing, as evidenced by a large body of work in gaze-based human-computer interaction, attentive user interfaces, and eye-based user modelling. However, current mobile eye trackers are still obtrusive, which not only makes them uncomfortable to wear and socially unacceptable in daily life, but also prevents them from being widely adopted in the social and behavioural sciences. To address these challenges we present InvisibleEye, a novel approach for mobile eye tracking that uses millimetre-size RGB cameras that can be fully embedded into normal glasses frames. To compensate for the cameras’ low image resolution of only a few pixels, our approach uses multiple cameras to capture different views of the eye, as well as learning-based gaze estimation to directly regress from eye images to gaze directions. We prototypically implement our system and characterise its performance on three large-scale, increasingly realistic, and thus challenging datasets: 1) eye images synthesised using a recent computer graphics eye region model, 2) real eye images recorded of 17 participants under controlled lighting, and 3) eye images recorded of four participants over the course of four recording sessions in a mobile setting. We show that InvisibleEye achieves a top person-specific gaze estimation accuracy of 1.79° using four cameras with a resolution of only 5×5 pixels. Our evaluations not only demonstrate the feasibility of this novel approach but, more importantly, underline its significant potential for finally realising the vision of invisible mobile eye tracking and pervasive attentive user interfaces.

   doi: 10.1145/3130971

   Paper: tonsen17_imwut.pdf

   @article{tonsen17_imwut, author = {Tonsen, Marc and Steil, Julian and Sugano, Yusuke and Bulling, Andreas}, title = {InvisibleEye: Mobile Eye Tracking Using Multiple Low-Resolution Cameras and Learning-Based Gaze Estimation}, journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)}, year = {2017}, doi = {10.1145/3130971}, volume = {1}, number = {3}, pages = {1--21} }

Here are some related news you might like to read next:

• Andreas Bulling joins the IMPRS-IS executive board
• SFB-TRR 161 funding period 3 approved
• SNSF Fellowship for Mihai Bace
• First visit by Hans Gellersen
• Eyes4ICU proposal accepted