Candidate
Luís Alberto Teixeira Aly

Date, Time and Location
July 22, 14:00, Sala de Atos, FEUP

President of the Jury
António Fernando Vasconcelos Cunha Castro Coelho (PhD), Associate Professor with Habilitation, Department of Informatics Engineering, Faculdade de Engenharia, Universidade do Porto.

Members
Javier Enrique Jaimovich Fernández (PhD), Associate Professor, Departamento de Sonido da Facultad de Artes, Universidad de Chile, Chile;
William Ruddock Primett (PhD), Postdoctoral Researcher, School of Digital Technologies, Tallinn University, Estónia;
Carla Maria de Jesus Montez Fernandes (PhD), Main Researcher, Instituto de Comunicação (ICNOVA), Faculdade de Ciências Sociais e Humanas, Universidade Nova de Lisboa;
Rui Pedro Amaral Rodrigues (PhD), Associate Professor, Department of Informatics Engineering, Faculdade de Engenharia, Universidade do Porto;
Gilberto Bernardes de Almeida (PhD), Assistant Professor, Department of Informatics Engineering, Faculdade de Engenharia, Universidade do Porto (Supervisor).

The thesis was co-supervised by Hugo Plácido da Silva (PhD), from Instituto Superior Técnico.

Abstract

This thesis, titled ’Emotion-driven Physiological Actor Dynamics For Interactive Theatre Sound,’ embarks on an exploratory journey into the innovative integration of physiological responses with emotional expression and sound design within theatre. This research investigates the intricate relationship between actors’ emotional states and physiological signals, delves into the impact of sound generated from physiological data on the actors’ emotional expression and agency, and examines how this novel integration can redefine traditional theatrical narratives and storytelling techniques. The study examines actors’ experiences and perceptions using qualitative and quantitative methodologies. It utilizes focus groups, observational studies, and sophisticated physiological sensors and surveys to capture and analyze the physiological signals and the feedback from actors. This approach allows for a nuanced understanding of the interplay between the physiological and emotional aspects of acting, shedding light on how actors embody and convey complex emotions through their performances. A key empirical contribution of this research is the DECEIVER dataset, which comprises extensive physiological recordings. These recordings provide valuable insights into the consistency and variability of emotional expression in performance settings. This dataset is a treasure trove for researchers and practitioners in the field, offering unprecedented detail and depth in understanding the physiological underpinnings of theatrical performance. Furthermore, the thesis presents a comprehensive historical analysis of the use of physiological sensors in interactive music, spanning the period from 1965 to 2023. This historical overview not only charts the technological evolution in this domain but also sets the stage for understanding the current trends and potential future developments. It contextualizes the research within the broader trajectory of technological advancements, highlighting the incremental and sometimes revolutionary changes that have shaped the current state of interactive music systems. The thesis introduces an empirical and functional taxonomy for Interactive Music Systems driven by physiological signals. This taxonomy represents a significant contribution to designing and applying physiological signals to interactive musical systems, providing a structured framework that can guide future developments in the field. It categorizes different approaches and methodologies in integrating physiological data into sound design, offering a comprehensive understanding of the potential and limitations of these systems. The research also involves the development of an extensive experimental protocol designed to analyze the physiological correlates of emotional valence and arousal in acting. A sophisticated software toolbox for data processing complements this protocol. The protocol’s design underscores the effectiveness of mental imagery in eliciting specific emotional states and highlights the complexity of emotional expression in theatre actors. This aspect of the research provides a methodological blueprint for future studies aiming to explore similar themes and questions. The Biosignal Processing Toolbox, a software tool for real-time operations integrating physiological signals with sound, is central to the study. The Biosignal Processing Toolbox enables the creation of dynamic, responsive soundscapes that interact with actors, enhancing the storytelling and engagement of the audience in the theatre. The toolbox is equipped to handle various physiological signals such as electromyography, electrocardiography, and electrodermal activity, each offering unique opportunities and challenges for sonification. The versatility of BarT lies in its ability to adapt and respond to different physiological inputs, making it an effective tool for sound designers in the theatre. A significant part of the research was a collaborative techno-artistic project, which utilized Samuel Beckett’s theatre as a backdrop. This project led to developing a prototype for an Interactive Music System driven by physiological sensors. This project explored the transformative possibilities of integrating physiological sensors and gesture typologies into theatre, providing fresh perspectives on character development and narrative construction. The project demonstrated the potential of this technology to bring a new dimension to theatrical performances, allowing for a more immersive and interactive experience for both actors and audiences. Despite its groundbreaking nature, the research acknowledges the challenges and limitations of such technological integrations. These include issues such as the need for real-time data processing, the necessity of actor-specific system calibration, technical and financial constraints, training requirements for actors and production teams, ensuring the comfort and unobtrusiveness of sensors during performances, ethical considerations related to the use of physiological data, and the subjective interpretation of such data in artistic contexts. In conclusion, this thesis contributes to theatre and interactive media art. Exploring the integration of physiological sensors in theatre sound design opens up new avenues for artistic expression and audience engagement. The development of the Biosignal Processing Toolbox and the DECEIVER dataset represent significant advancements in the field, paving the way for more immersive, interactive, and expressive forms of storytelling. This research provides novel perspectives for sound design and actor training and contributes to the broader discourse on the intersection of technology and art.

Candidate
Maria José Gomes Pedroto

Date, Time and Location
July 22, 10:00, Sala de Atos of FEUP

President of the Jury
Carlos Miguel Ferraz Baquero-Moreno (PhD), Full Professor, Faculdade de Engenharia, Universidade do Porto

Members
Myra Spiliopoulou (PhD), Full Professor of Business Information Systems da Faculty of Computer Science da Otto-von-Guericke-University Magdeburg, Alemanha;
Manuel Filipe Vieira Torres dos Santos (PhD), Associate Professor with Habilitation, Department of Information Systems, Escola de Engenharia, Universidade do Minho;
Alípio Mário Guedes Jorge (PhD), Full Professor, Department of Computer Science, Faculdade de Ciências, Universidade do Porto (Supervisor);
Rui Carlos Camacho de Sousa Ferreira da Silva (PhD), Associate Professor, Department of Informatics Engineering, Faculdade de Engenharia, Universidade do Porto.

The thesis was co-supervised by João Pedro Carvalho Leal Mendes Moreira (PhD), Associate Professor in the Department of Informatics Engineering, Faculdade de Engenharia, Universidade do Porto.

Abstract

This work is centered on modeling and predicting Time-to-Event (TTE) episodes, and has two distinct purposes. The first purpose is to explore the usage of genealogical data for time to event prediction. Additionally, this work aims to aid medical professionals in improving the diagnosis and prognosis of patients afflicted with Hereditary Transthyretin Amyloidosis (ATTRv amyloidosis). This is a genetic disease with a strong historical background in the fishing villages of Póvoa do Varzim, northern Portugal. In order to explore the value of genealogical data for time-to-event prediction, this work has contributions in feature engineering, namely within the area of feature construction and selection. To this end, it explores and compares a summarizing approach focused on manually extracting meaningful features from genealogical trees with a more automated one using embeddings. It contributes to model construction by creating a multivariate data-oriented approach that tracks a patient’s risk of developing disease onset through different ages. It also explores the impact of combining different age models of neighboring time windows. Finally, it contributes to model evaluation by addressing the implementation issues of a business based

approach to evaluate the expected return of changing clinical guidelines. It also presents robust evaluation schemas that assist the multivariate data-oriented approach in selecting the best model. The application is focused on patients with ATTRv Amyloidosis. To present and characterize the work done, this thesis is structured into four main sections. It begins with an introduction and a presentation of ATTRv Amyloidosis from a medically historic perspective. Then it presents the relevant background by dwelling into the connection of time to event prediction with feature engineering, model construction and model evaluation, as well as introducing key concepts of genealogical studies. After this, it presents its technical contributions, in the form of the main publications that constitute this work (one paper by chapter). It ends with an epilogue section which overviews the work performed, shares the main conclusions, and, finally, discusses the thesis from a technical and clinical perspective.

Keywords: Time-to-Event Data; Data Modeling; Regression Models

Candidate:
Mariana de Oliveira Magalhães

Date, Time and Location:
July 8, 10:00, Sala de Atos (I-105)  DEEC, FEUP

President of the Jury:
João Manuel Paiva Cardoso, PhD, Full Professor, Departamento de Engenharia Informática, Faculdade de Engenharia, Universidade do Porto.

Members:
Alfredo Manuel dos Santos Ferreira Júnior, PhD, Associate Professor, Departamento de Engenharia Informática, Instituto Superior Técnico da Universidade de Lisboa;
Maria Beatriz Alves de Sousa Santos, PhD, Associate Professor with habilitation, Departamento de Eletrónica, Telecomunicações e Informática, Universidade de Aveiro;
Mário Sérgio Carvalho Teixeira, PhD, Assistant Professor, Departamento de Economia, Sociologia e Gestão, Universidade de Trás-os-Montes e Alto Douro;
António Fernando Vasconcelos Cunha Castro Coelho, PhD, Associate Professor with habilitation, Departamento de Engenharia Informática, Faculdade de Engenharia, Universidade do Porto (Supervisor);
António Augusto de Sousa, PhD, Associate Professor, Departamento de Engenharia Informática, Faculdade de Engenharia, Universidade do Porto.

The thesis was co-supervised by Doutor Maximino Bessa, Associate Professor with Habilitation, Universidade de Trás-os-Montes e Alto Douro.

Abstract
This thesis aims to explore how multisensory experiences in virtual reality influence users’ emotional responses and the vividness of their mental imagery, focusing on the impact of the demographics of gender and age. This thesis is driven by the recent rapid development of virtual reality in tourism, characterized by increasingly immersive multisensory experiences. It addresses the knowledge gap related to the limited understanding of how multisensory stimuli impact users’ emotional responses and their mental imagery ability, considering the particular case of virtual tourism.
Two immersive virtual experiences were developed for this purpose. Multisensory combinations of visual, auditory, haptic, olfactory, and taste stimuli were strategically integrated at specific stages of the two experiments, after being previously validated in a focus group session. One of these virtual experiences included a scenario intended to elicit positive emotions in the user by resorting to a selection of pleasant multisensory stimuli, designated as the “positive IVE” (positive Immersive Virtual Environment). The other experience sought the contrary: to induce negative emotions in the user through a combination of unpleasant multisensory stimuli, which was labeled as the “negative IVE” (negative Immersive Virtual Environment). The basic combination of visual and auditory stimuli was consistently used during the entire experiment. Additional stimuli – taste, haptic, and smell – were introduced one by one, sequentially. Finally, all these stimuli were combined for a comprehensive experience. A between-subjects experimental design was developed to explore and compare the users’ emotional responses and vividness of visual imagery after each stimuli combination in the two virtual experiences, resorting to in-VR questionnaires for data collection. Key findings include the impact of different positive and negative multisensory stimuli combinations on the users’ emotional responses, and how they, in turn, influence mental imagery. This research further suggests an inverse relationship between the intensity of the user’s emotions and their mental imagery ability. Nevertheless, neither age nor gender was found to influence this relationship in either the positive or negative scenarios. Additionally, this investigation provides insights into the specific emotions triggered by the used multisensory stimuli combinations, addressing a need long identified by various researchers in the field. This thesis contributes to understanding multisensory stimuli in virtual reality, highlighting its potential for application in various fields. It provides insights for future research in creating user-centered virtual reality tourism applications and understanding individual differences.

Keywords: Multisensory Virtual Reality; Virtual Reality; Emotional Responses; Vividness of Mental Imagery; Virtual Tourism.

Candidate
Diogo Fernandes Santos

Date, Time and Location  
June 25, 14:30, Sala de Atos FEUP

President of the Jury
António Fernando Vasconcelos Cunha Castro Coelho, PhD, Associate Professor with habilitation, Faculdade de Engenharia, Universidade do Porto.

Vogais
Lynn Rosalina Gama Alves, PhD, Associate Professor, Instituto de Humanidades, Artes e Ciências Professor Milton Santos, Universidade Federal da Bahia, Brasil;
António Maria Salvado Coxito Granado, PhD, Associate Professor, Departamento de Ciências da Comunicação, Faculdade de Ciências Sociais e Humanas, Universidade Nova de Lisboa;
Liliana Filipa Vale Costa, PhD, Associate Professor, Departamento de Comunicação e Arte, Universidade de Aveiro;
Cátia Ferreira, PhD, Assistant Professor, Faculdade de Ciências Humanas, Universidade Católica Portuguesa;
João Carlos de Matos Paiva, PhD, Associate Professor with Habilitation, Departamento de Química e Bioquímica, Faculdade de Ciências, Universidade do Porto;
Carla Susana Lopes Morais, PhD, Assistant Professor with Habilitation, Departamento de Química e Bioquímica, Faculdade de Ciências, Universidade do Porto (Supervisor).

The thesis was co-supervised by Professor Nelson Troca Zagalo, Full Professor at the Universidade de Aveiro.

Abstract
The adoption of multimedia tools and objects came with challenges and opportunities for everyone involved in science communication and science learning initiatives. People interacting with the content, with the producer, or even the transformation of the consumer in a producer. Participation and dialogue become staples in science communication, with scientists and research units having to embrace this brave, but not so new world. In this experience, transmedia served as a guideline to think about science communication from within a research unit in the multimedia space.
In this work that was designed within Chemistry Research Unit of University of Porto (CIQUP) and shaped by ideas like participation, collaboration, dialogue, and interactivity, scientists were invited to co-create a fun serious video game to communicate their science (nanoparticles for drug delivery). To combine a serious portrait of science with fun elements that are typical of games, scientists were crucial to define theme, systems, mechanics, and narrative. It was developed a prototype (Nano Entregas) aimed to communicate about nanoparticles for drug delivery. The prototype was played by 170 participants from the 7th to the 11th school level in an experience aimed to measure the potential of the tool in two domains: as a science communication object capable of transmitting knowledge about nanoparticles and other scientific concepts associated with the subject; and as a game. To do so, data was collected through the implementation of a questionnaire and through the application of a serious game evaluation scale. The video game was also played by researchers and chemistry teachers, from whom observations were collected. Before the core gaming experience, the prototype was tested with the target audience (27 of the 170 participants), and three participants with no previous connection with the project (one teacher, one researcher, and one doctor). The prototype developed has shown potential to be used in multiple scenarios (formal and informal) where nanoparticles can be a subject to present. Most players (from the 170 students) have captured the basic information that was delivered. One was the metaphor used to introduce the size of nanoparticles within the real world scale of things, with 66,5% of the participants being able to deduce it from the bits of dialogue between the in-game characters. The percentage of correct answers were also higher in the three scientific questions: 85,3%, 78,2%, and 58,2%, respectively. The prototype was well received by the audience, with the 11 Factors in the scale collecting positive notes. The experience shows that scientists are eager to collaborate in challenging and innovative strategies to communicate with the audience, with the metaphor that shaped the prototype in terms of theme, narrative, systems, and mechanics, emerging directly from their contribution. The prototype shows that science can be portraited with seriousness in a video game that aimed for fun. Dialogue can be key to quickly find a common ground between researchers and communicators or developers, which then can result in the development of engaging tools to communicate about complex topics. Younger generations understand video games’ language. And, through it, scientists, communicators, teachers, and parents, can present and share scientific knowledge in a fun and interactive learning exercise.

Keywords: science communication; game development; transmedia; research units; chemistry; nanoparticles.