Regresar

Investigation of the Activities Performed by Experimental Researchers in a Software Engineering Lab Using an Ethnographic Methodology

Abstract:

Context: Replication plays a critical role in building cumulative scientific knowledge. However, in the context of Empirical Software Engineering (ESE), replication efforts face persistent difficulties, both technical and methodological, which hinder reproducibility and generalizability. We explored whether the problems were due to formal issues, such as under-specification or miscommunication, or intrinsic reasons, i.e., the existing replication procedures may not meet the researchers’ needs. Objective: To understand how ESE researchers conduct experiments in real-world settings. Method: We conducted an ethnographic study with an experimental software engineering group, using interviews, observations, and analysis of internal documentation. Results: We have created conceptual and process models representing experimentation, replication, and synthesis in the target research group. These models align with mainstream procedures at a high level but often break down in practice during specialized tasks such as coordination or documentation. The experimental process observed in the group differs from common assumptions and textbook descriptions in terms of (1) the number and diversity of activities involved, (2) the presence of differentiated roles, (3) the granularity of conceptual elements, and (4) the varying perspectives across subareas or families of experiments. Conclusions: The discrepancies between actual laboratory practices and idealized process models may hinder knowledge transfer and complicate replication. We plan to extend this research by involving additional research groups, for instance, through surveys, to examine the generalizability of our findings.