AuthorsC. Midoglu, A. Storås, S. S. Sabet, M. Hammou, S. Hicks, I. Strümke, M. Riegler, C. Griwodz and P. Halvorsen
TitleExperiences and Lessons Learned from a Crowdsourced-Remote Hybrid User Survey Framework
AfilliationMachine Learning
Project(s)Department of Holistic Systems
Publication TypeProceedings, refereed
Year of Publication2022
Conference Name2022 IEEE International Symposium on Multimedia (ISM)
Place PublishedItaly

Subjective user studies are important to ensure the fidelity and usability of systems that generate multimedia content. Testing how end-users and domain experts perceive multimedia assets might provide crucial information. In this paper, we present our experiences with the open source hybrid crowdsourced-remote user survey framework called Huldra, which is intended for conducting web-based subjective user studies and aims to integrate the individual benefits associated with traditional, crowdsourced, and remote methods. We disseminate our experiences and insights from two actively deployed use cases and discuss challenges and opportunities associated with using Huldra as a framework for conducting user studies.

Citation Key43138

Contact person