|Authors||A. Eichhorn, P. Ni and R. Eg|
|Editors||D. C. A. Bulterman|
|Title||Randomised Pair Comparison - an Economic and Robust Method for Audiovisual Quality Assessment|
|Afilliation||Communication Systems, Communication Systems|
|Publication Type||Proceedings, refereed|
|Year of Publication||2010|
|Conference Name||The 20th International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 2010)|
Subjective quality perception studies with human observers are essential for multimedia system design. Such studies are known to be expensive and difficult to administer. They re- quire time, a detailed knowledge of experimental designs and a level of control which can often only be achieved in a lab- oratory setting. Hence, only very few researchers consider running subjective studies at all. In this paper we present Randomised Pair Comparison (r/pc), an easy-to-use, flexible, economic and robust exten- sion to conventional pair comparison methods. r/pc uses random sampling to select a unique and small subset of pairs for each assessor, thus separating session duration from the experimental design. With r/pc an experimenter can freely define the duration of sessions and balance between costs and accuracy of an experiment. On a realistic example study we show that r/pc is able to create stable results with an accuracy close to full factorial designs, yet much lower costs. We also provide initial evi- dence that r/pc can avoid unpleasant fatigue and learning effects which are common in long experiment sessions.