AuthorsA. Eichhorn, P. Ni and R. Eg
EditorsD. C. A. Bulterman
TitleRandomised Pair Comparison - an Economic and Robust Method for Audiovisual Quality Assessment
AfilliationCommunication Systems, Communication Systems
Publication TypeProceedings, refereed
Year of Publication2010
Conference NameThe 20th International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 2010)
ISBN Number978-1-4503-0043-8

Subjective quality perception studies with human observers are essential for multimedia system design. Such studies are known to be expensive and difficult to administer. They re- quire time, a detailed knowledge of experimental designs and a level of control which can often only be achieved in a lab- oratory setting. Hence, only very few researchers consider running subjective studies at all. In this paper we present Randomised Pair Comparison (r/pc), an easy-to-use, flexible, economic and robust exten- sion to conventional pair comparison methods. r/pc uses random sampling to select a unique and small subset of pairs for each assessor, thus separating session duration from the experimental design. With r/pc an experimenter can freely define the duration of sessions and balance between costs and accuracy of an experiment. On a realistic example study we show that r/pc is able to create stable results with an accuracy close to full factorial designs, yet much lower costs. We also provide initial evi- dence that r/pc can avoid unpleasant fatigue and learning effects which are common in long experiment sessions.

Citation KeySimula.mpg.8