Authors | A. Eichhorn, P. Ni and R. Eg |
Editors | D. C. A. Bulterman |
Title | Randomised Pair Comparison - an Economic and Robust Method for Audiovisual Quality Assessment |
Afilliation | Communication Systems, Communication Systems |
Status | Published |
Publication Type | Proceedings, refereed |
Year of Publication | 2010 |
Conference Name | The 20th International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 2010) |
Pagination | 63-68 |
Publisher | ACM |
ISBN Number | 978-1-4503-0043-8 |
Abstract | Subjective quality perception studies with human observers are essential for multimedia system design. Such studies are known to be expensive and difficult to administer. They re- quire time, a detailed knowledge of experimental designs and a level of control which can often only be achieved in a lab- oratory setting. Hence, only very few researchers consider running subjective studies at all. In this paper we present Randomised Pair Comparison (r/pc), an easy-to-use, flexible, economic and robust exten- sion to conventional pair comparison methods. r/pc uses random sampling to select a unique and small subset of pairs for each assessor, thus separating session duration from the experimental design. With r/pc an experimenter can freely define the duration of sessions and balance between costs and accuracy of an experiment. On a realistic example study we show that r/pc is able to create stable results with an accuracy close to full factorial designs, yet much lower costs. We also provide initial evi- dence that r/pc can avoid unpleasant fatigue and learning effects which are common in long experiment sessions. |
Citation Key | Simula.mpg.8 |