AuthorsA. Karahasanovic, U. N. Hinkel, D. I. K. Sjøberg, A. K. Levine and R. Thomas
TitleA Controlled Experiment to Evaluate Feedback-Collection Tool in Software Engineering Research, Technical Report 2004-8, Simula Research Laboratory
Publication TypeTechnical reports
Year of Publication2004
PublisherSimula Research Laboratory

Objective: To compare the feedback-collection method with think-aloud methods regarding effects on subjects' performance and type of collected information in software engineering experiments. Rationale: Think-aloud methods provide a rich set of data about experiments but are time-consuming in data collection. Alternatively, subjects can be interrupted during experiments and asked for written feedback. However, there has been no systematic comparison of these methods. Design of study: We conducted a controlled experiment to evaluate three methods for qualitative data collection: feedback-collection, concurrent think-aloud and retrospective think-aloud. Results: There is no significant difference in subjects' performance for the three data collection methods. The types of information gathered by these three methods differed; in particular, the feedback collection method provided more explanations about the problem-solving process and more information about subjects' perceptions of the experiment. Conclusions: This research confirms that the feedback-collection method is an appropriate data collection method in software engineering experiments.


Technical Report 2004-8, Simula Research Laboratory

Citation KeyKarahasanovic.2004.1