AuthorsH. Svoren, V. Thambawita, P. Halvorsen, P. Jakobsen, E. Garcia-Ceja, F. M. Noori, H. L. Hammer, M. Lux, M. Riegler and S. Hicks
TitleToadstool: A Dataset for Training Emotional IntelligentMachines Playing Super Mario Bros
AfilliationMachine Learning
Project(s)Department of Holistic Systems
Publication TypeProceedings, refereed
Year of Publication2020
Conference NameThe ACM Multimedia Systems Conference (MMSys)
Place PublishedThe ACM Multimedia Systems Conference (MMSys)

Games are often defined as engines of experience, and they are heavily relying on emotions, they arouse in players. In this paper, we present a dataset called Toadstool as well as a reproducible methodology to extend on the dataset. The dataset consists of video, sensor, and demographic data collected from ten participants playing Super Mario Bros, an iconic and famous video game. The sensor data is collected through an Empatica E4 wristband, which provides high-quality measurements and is graded as a medical device. In addition to the dataset and the methodology for data collection, we present a set of baseline experiments which show that we can use video game frames together with the facial expressions to predict the blood volume pulse of the person playing Super Mario Bros. With the dataset and the collection methodology we aim to contribute to research on emotionally aware machine learning algorithms, focusing on reinforcement learning and multimodal data fusion. We believe that the presented dataset can be interesting for a manifold of researchers to explore exciting new interdisciplinary questions. 

Citation Key27331