AuthorsK. Petersen and J. Molléri
EditorsR. Ali, H. Kaindl and L. Maciaszek
TitlePreliminary Evaluation of a Survey Checklist in the Context of Evidence-based Software Engineering Education
AfilliationSoftware Engineering
Project(s)Department of IT Management
Publication TypeProceedings, refereed
Year of Publication2021
Conference Name16th International Conference on Evaluation of Novel Approaches to Software EngineeringProceedings of the 16th International Conference on Evaluation of Novel Approaches to Software Engineering
PublisherSCITEPRESS - Science and Technology Publications
Place PublishedOnline Streaming, --- Select a Country ---
ISBN Number978-989-758-508-1
KeywordsChecklist, One-group Quasi-experiment, Students, survey

Background: In order to judge evidence it is important to be able to assess study quality. Checklists are means to objectify the assessment. In an earlier study we proposed and evaluated a checklist for surveys, which was assessed by experts. Objective: (1) To assess whether the use of the checklist enables students with limited experience in research to consistently and accurately assess the quality of a research paper. (2) To elicit qualitative feedback to identify improvements to the checklist. Method: The students reviewed a survey in a one-group posttest-only quasi-experiment using the checklist. In total 13 students participated in the context of the course Evidence-based software engineering as part of the study program Information Systems at Flensburg University of Applied Sciences. Results: In total the students achieved 74% percent of agreement among each other. However, the Kappa values indicated mostly a poor level of agreement considering the classification by Fleiss. In addition, the students were quite inaccurate assessing the questions. Though, they performed well on questions for research objectives and the identification of population. Conclusion: Findings indicate that students do not assess reliably. However, further investigations are needed to substantiate the findings.

Citation Key27836

Contact person