Candidate:
Lázaro Gabriel Barros da Costa
Date, Time and Location:
18th of July 2025, 16:00, Sala de Atos of the Faculty of Engineering of University of Porto.
President of the Jury:
Pedro Nuno Ferreira da Rosa da Cruz Diniz (PhD), Full Professor, Department of Informatics Engineering, Faculty of Engineering, University of Porto.
Members:
Tanu Malik (PhD), Associate Professor, Department of Electrical Engineering and Computer Science, University of Missouri, U.S.A;
Miguel Carlos Pacheco Afonso Goulão (PhD), Associate Professor, Department of Computer Science, Faculty of Science and Technology, New University of Lisbon;
Gabriel de Sousa Torcato David (PhD), Associate Professor, Department of Informatics Engineering, Faculty of Engineering, University of Porto;
Jácome Miguel Costa da Cunha (PhD), Associate Professor, Department of Informatics Engineering, Faculty of Engineering, University of Porto (Supervisor).
The thesis was co-supervised by Susana Alexandra Tavares Meneses Barbosa (PhD), Senior Researcher at INESCTEC Porto.
Abstract:
Scientific reproducibility and replicability are essential pillars of credible research, especially as computational experiments become increasingly prevalent across diverse scientific disciplines such as chemistry, climate science, and biology. Despite strong advocacy for Open Science and adherence to FAIR (Findable, Accessible, Interoperable, and Reusable) principles, achieving true reproducibility remains a formidable challenge for many researchers. Key issues such as complex dependency management, inadequate metadata, and the often cumbersome access to necessary code and data severely hamper reproducibility efforts. Moreover, existing reproducibility tools frequently offer piecemeal solutions that fail to address the multifaceted needs of diverse and complex experimental setups, particularly those that span multiple programming languages and involve intricate data systems. This thesis addresses these challenges by presenting a comprehensive framework designed to enhance computational reproducibility across a variety of scientific fields. Our approach involved a detailed systematic review of existing reproducibility tools to identify prevailing gaps and limitations in their design and functionality. This review underscored the fragmented nature of these tools, each supporting only aspects of the reproducibility process but none providing a holistic solution, particularly for experiments that require robust data handling or support for many programming languages.
To bridge these gaps, we introduced SCIREP, an innovative framework that automates essential aspects of the reproducibility workflow such as dependency management, containerization, and cross platform compatibility. This framework was rigorously evaluated using a curated dataset of computational experiments, achieving a reproducibility success rate of 94%.
Furthering the accessibility and usability of reproducible research, we developed SCICONV, a conversational interface that simplifies the configuration and execution of computational experiments using natural language processing. This interface significantly reduces the technical barriers traditionally associated with setting up reproducible studies, allowing researchers to interact with the system through simple, guided conversations. Evaluation results indicated that SCICONV successfully reproduced 83% of the experiments in our curated dataset with minimal user input, highlighting its potential to make reproducible research more accessible to a broader range of researchers.
Moreover, recognizing the critical role of user studies in evaluating tools, methodologies, and prototypes, particularly in software engineering and behavioral sciences, this thesis also extends into the realm of experimental tool evaluation. We conducted a thorough analysis of existing tools used for software engineering and behavioral science experiments, identifying and proposing specific features designed to enhance their functionality and ease of use for conducting user studies. These proposed features were validated through a survey involving the research community, confirming their relevance and the need for their integration into existing and future tools. The contributions of this thesis are manifold, encompassing the development of a classification framework for reproducibility tools, the creation of a standardized benchmark dataset for assessing tool efficacy, and the formulation of SCIREP and SCICONV to significantly advance the state-of-the-art in computational reproducibility. Looking forward, the research will focus on expanding the capabilities of reproducibility tools to support more complex scientific workflows, further enhancing user interfaces, and integrating additional functionalities to fully support user studies. By doing so, this work aims to pave the way for a more robust, accessible, and efficient computational reproducibility ecosystem that can meet the evolving needs of the global research community.
Keywords: Reproducibility; Replicability; Reusability; Computational Experiments; Conversational User Interface; User Studies.