Affective State Prediction from Smartphone Touch and Sensor Data in the Wild

Rafael Wampfler, Severin Klingler, Barbara Solenthaler, Victor R. Schinazi, Markus Gross, Christian Holz

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

Abstract

Knowledge of users' affective states can improve their interaction with smartphones by providing more personalized experiences (e.g., search results and news articles). We present an affective state classification model based on data gathered on smartphones in real-world environments. From touch events during keystrokes and the signals from the inertial sensors, we extracted two-dimensional heat maps as input into a convolutional neural network to predict the affective states of smartphone users. For evaluation, we conducted a data collection in the wild with 82 participants over 10 weeks. Our model accurately predicts three levels (low, medium, high) of valence (AUC up to 0.83), arousal (AUC up to 0.85), and dominance (AUC up to 0.84). We also show that using the inertial sensor data alone, our model achieves a similar performance (AUC up to 0.83), making our approach less privacy-invasive. By personalizing our model to the user, we show that performance increases by an additional 0.07 AUC.

Original languageEnglish
Title of host publicationCHI' 22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
EditorsSimone Barbosa, Cliff Lampe, Caroline Appert, David A. Shamma, Steven Drucker, Julie Williamson, Koji Yatani
Place of PublicationNew York
PublisherAssociation for Computing Machinery (ACM)
ISBN (Electronic)9781450391573
DOIs
Publication statusPublished - 29 Apr 2022
EventCHI 2022 - Conference on Human Factors in Computing Systems - New Orleans, United States
Duration: 30 Apr 20225 May 2022
https://chi2022.acm.org/

Publication series

NameConference on Human Factors in Computing Systems - Proceedings

Conference

ConferenceCHI 2022 - Conference on Human Factors in Computing Systems
Abbreviated titleCHI
Country/TerritoryUnited States
CityNew Orleans
Period30/04/225/05/22
Internet address

Fingerprint

Dive into the research topics of 'Affective State Prediction from Smartphone Touch and Sensor Data in the Wild'. Together they form a unique fingerprint.

Cite this