|Authors||A. Karahasanovic, U. N. Hinkel, D. I. K. Sjøberg, A. K. Levine and R. Thomas|
|Title||A Controlled Experiment to Evaluate Feedback-Collection Tool in Software Engineering Research, Technical Report 2004-8, Simula Research Laboratory|
|Publication Type||Technical reports|
|Year of Publication||2004|
|Publisher||Simula Research Laboratory|
Objective: To compare the feedback-collection method with think-aloud methods regarding effects on subjects' performance and type of collected information in software engineering experiments. Rationale: Think-aloud methods provide a rich set of data about experiments but are time-consuming in data collection. Alternatively, subjects can be interrupted during experiments and asked for written feedback. However, there has been no systematic comparison of these methods. Design of study: We conducted a controlled experiment to evaluate three methods for qualitative data collection: feedback-collection, concurrent think-aloud and retrospective think-aloud. Results: There is no significant difference in subjects' performance for the three data collection methods. The types of information gathered by these three methods differed; in particular, the feedback collection method provided more explanations about the problem-solving process and more information about subjects' perceptions of the experiment. Conclusions: This research confirms that the feedback-collection method is an appropriate data collection method in software engineering experiments.
Technical Report 2004-8, Simula Research Laboratory