Candidate
Tiago André Queiroz Soares da Costa

Date, Time and Location:
September 16, 14:30, Sala de Atos da Faculdade de Engenharia da Universidade do Porto

President of the Jury:
Doutor Carlos Miguel Ferraz Baquero-Moreno, Professor Catedrático da Faculdade de Engenharia da Universidade do Porto

Members:
Klara Nahrstedt, PhD, Full Professor, Department of Computer Science, University of Illinois at Urbana-Champaign, United States of America;
Pedro António Amado de Assunção, PhD, Coordinator Professor, Departamento de Engenharia Eletrotécnica, Escola Superior de Tecnologia e Gestão do Instituto Politécnico de Leiria;
Luís António Pereira de Meneses Corte-Real, PhD, Associate Professor, Departamento de Engenharia Eletrotécnica e de Computadores, Faculdade de Engenharia da Universidade do Porto;
Maria Teresa Magalhães da Silva Pinto de Andrade, PhD, Assistant Professor, Departamento de Engenharia Eletrotécnica e de Computadores, Faculdade de Engenharia da Universidade do Porto (Supervisor).

Abstract:
This thesis proposes a novel approach to immersive, multiview media distribution that uses Deep Learning models and user-centric data to predict user interest in the near future while multimediacontent is being presented. The main objective of this thesis is to give the user a truly ubiquitousmultimedia immersive experience without the need for expensive equipment, while also allowing him or her to see the scene being presented on the screen from almost any angle, as if they were actually there when the scene was shot. A methodological approach was envisioned based on the literature review and the identification of gaps in immersive streaming scenarios, which resulted in the conceptualization of a brand new architecture that was coined Smooth Multiview (SmoothMV). This architecture is capable of analysing user behaviour data in real-time and preemptively preparing content delivery accordingly based on viewing interests demonstrated by users while visualising a particular scene. Users effortlessly provide behaviour data without equiring intrusive equipment, which is then processed using the novel concept of the Hot&Cold matrix, which this thesis describes. With the use of this concept, the screen is divided into nine separate regions, each of which is connected to a neighbouring view that the SmoothMV architecture can prepare and choose to present. Separate queues designated for playback and buffering of upcoming content segments are introduced to provide minimal delay without compromising the user experience, since content adaptation is closely linked to user inputs. The number of views that are available and the approach employed for analysing behaviour while viewing content and selecting which view should be processed in the following moment affect how these queues are managed. This thesis developed from a purely reactive approach to a sophisticated, twofold Deep Learning architecture that can accurately identify the view that best fits the interests of the user with a high degree of accuracy. The development of a new dataset was needed in order to achieve this level of performance, as the data provided by existing datasets was not suitable for the scenario that was proposed. After a series of 128 experiments were conducted to collect visual attention data from 45 participants while viewing multi-perspective content, the Data2MV dataset was created and made available to the public. This thesis’ fundamental concepts and practical outputs are considered to be of significant importance to the body of knowledge currently available in the field of research, while also offering relevant tools for the general enhancement of current content distribution architectures.

Keywords: Multimedia, Streaming, Multiview, Focus-of-Attention, Deep Learning

Candidate
Luís Alberto Teixeira Aly

Date, Time and Location
July 22, 14:00, Sala de Atos, FEUP

President of the Jury
António Fernando Vasconcelos Cunha Castro Coelho (PhD), Associate Professor with Habilitation, Department of Informatics Engineering, Faculdade de Engenharia, Universidade do Porto.

Members
Javier Enrique Jaimovich Fernández (PhD), Associate Professor, Departamento de Sonido da Facultad de Artes, Universidad de Chile, Chile;
William Ruddock Primett (PhD), Postdoctoral Researcher, School of Digital Technologies, Tallinn University, Estónia;
Carla Maria de Jesus Montez Fernandes (PhD), Main Researcher, Instituto de Comunicação (ICNOVA), Faculdade de Ciências Sociais e Humanas, Universidade Nova de Lisboa;
Rui Pedro Amaral Rodrigues (PhD), Associate Professor, Department of Informatics Engineering, Faculdade de Engenharia, Universidade do Porto;
Gilberto Bernardes de Almeida (PhD), Assistant Professor, Department of Informatics Engineering, Faculdade de Engenharia, Universidade do Porto (Supervisor).

The thesis was co-supervised by Hugo Plácido da Silva (PhD), from Instituto Superior Técnico.

Abstract

This thesis, titled ’Emotion-driven Physiological Actor Dynamics For Interactive Theatre Sound,’ embarks on an exploratory journey into the innovative integration of physiological responses with emotional expression and sound design within theatre. This research investigates the intricate relationship between actors’ emotional states and physiological signals, delves into the impact of sound generated from physiological data on the actors’ emotional expression and agency, and examines how this novel integration can redefine traditional theatrical narratives and storytelling techniques. The study examines actors’ experiences and perceptions using qualitative and quantitative methodologies. It utilizes focus groups, observational studies, and sophisticated physiological sensors and surveys to capture and analyze the physiological signals and the feedback from actors. This approach allows for a nuanced understanding of the interplay between the physiological and emotional aspects of acting, shedding light on how actors embody and convey complex emotions through their performances. A key empirical contribution of this research is the DECEIVER dataset, which comprises extensive physiological recordings. These recordings provide valuable insights into the consistency and variability of emotional expression in performance settings. This dataset is a treasure trove for researchers and practitioners in the field, offering unprecedented detail and depth in understanding the physiological underpinnings of theatrical performance. Furthermore, the thesis presents a comprehensive historical analysis of the use of physiological sensors in interactive music, spanning the period from 1965 to 2023. This historical overview not only charts the technological evolution in this domain but also sets the stage for understanding the current trends and potential future developments. It contextualizes the research within the broader trajectory of technological advancements, highlighting the incremental and sometimes revolutionary changes that have shaped the current state of interactive music systems. The thesis introduces an empirical and functional taxonomy for Interactive Music Systems driven by physiological signals. This taxonomy represents a significant contribution to designing and applying physiological signals to interactive musical systems, providing a structured framework that can guide future developments in the field. It categorizes different approaches and methodologies in integrating physiological data into sound design, offering a comprehensive understanding of the potential and limitations of these systems. The research also involves the development of an extensive experimental protocol designed to analyze the physiological correlates of emotional valence and arousal in acting. A sophisticated software toolbox for data processing complements this protocol. The protocol’s design underscores the effectiveness of mental imagery in eliciting specific emotional states and highlights the complexity of emotional expression in theatre actors. This aspect of the research provides a methodological blueprint for future studies aiming to explore similar themes and questions. The Biosignal Processing Toolbox, a software tool for real-time operations integrating physiological signals with sound, is central to the study. The Biosignal Processing Toolbox enables the creation of dynamic, responsive soundscapes that interact with actors, enhancing the storytelling and engagement of the audience in the theatre. The toolbox is equipped to handle various physiological signals such as electromyography, electrocardiography, and electrodermal activity, each offering unique opportunities and challenges for sonification. The versatility of BarT lies in its ability to adapt and respond to different physiological inputs, making it an effective tool for sound designers in the theatre. A significant part of the research was a collaborative techno-artistic project, which utilized Samuel Beckett’s theatre as a backdrop. This project led to developing a prototype for an Interactive Music System driven by physiological sensors. This project explored the transformative possibilities of integrating physiological sensors and gesture typologies into theatre, providing fresh perspectives on character development and narrative construction. The project demonstrated the potential of this technology to bring a new dimension to theatrical performances, allowing for a more immersive and interactive experience for both actors and audiences. Despite its groundbreaking nature, the research acknowledges the challenges and limitations of such technological integrations. These include issues such as the need for real-time data processing, the necessity of actor-specific system calibration, technical and financial constraints, training requirements for actors and production teams, ensuring the comfort and unobtrusiveness of sensors during performances, ethical considerations related to the use of physiological data, and the subjective interpretation of such data in artistic contexts. In conclusion, this thesis contributes to theatre and interactive media art. Exploring the integration of physiological sensors in theatre sound design opens up new avenues for artistic expression and audience engagement. The development of the Biosignal Processing Toolbox and the DECEIVER dataset represent significant advancements in the field, paving the way for more immersive, interactive, and expressive forms of storytelling. This research provides novel perspectives for sound design and actor training and contributes to the broader discourse on the intersection of technology and art.

Candidate
Maria José Gomes Pedroto

Date, Time and Location
July 22, 10:00, Sala de Atos of FEUP

President of the Jury
Carlos Miguel Ferraz Baquero-Moreno (PhD), Full Professor, Faculdade de Engenharia, Universidade do Porto

Members
Myra Spiliopoulou (PhD), Full Professor of Business Information Systems da Faculty of Computer Science da Otto-von-Guericke-University Magdeburg, Alemanha;
Manuel Filipe Vieira Torres dos Santos (PhD), Associate Professor with Habilitation, Department of Information Systems, Escola de Engenharia, Universidade do Minho;
Alípio Mário Guedes Jorge (PhD), Full Professor, Department of Computer Science, Faculdade de Ciências, Universidade do Porto (Supervisor);
Rui Carlos Camacho de Sousa Ferreira da Silva (PhD), Associate Professor, Department of Informatics Engineering, Faculdade de Engenharia, Universidade do Porto.

The thesis was co-supervised by João Pedro Carvalho Leal Mendes Moreira (PhD), Associate Professor in the Department of Informatics Engineering, Faculdade de Engenharia, Universidade do Porto.

Abstract

This work is centered on modeling and predicting Time-to-Event (TTE) episodes, and has two distinct purposes. The first purpose is to explore the usage of genealogical data for time to event prediction. Additionally, this work aims to aid medical professionals in improving the diagnosis and prognosis of patients afflicted with Hereditary Transthyretin Amyloidosis (ATTRv amyloidosis). This is a genetic disease with a strong historical background in the fishing villages of Póvoa do Varzim, northern Portugal. In order to explore the value of genealogical data for time-to-event prediction, this work has contributions in feature engineering, namely within the area of feature construction and selection. To this end, it explores and compares a summarizing approach focused on manually extracting meaningful features from genealogical trees with a more automated one using embeddings. It contributes to model construction by creating a multivariate data-oriented approach that tracks a patient’s risk of developing disease onset through different ages. It also explores the impact of combining different age models of neighboring time windows. Finally, it contributes to model evaluation by addressing the implementation issues of a business based

approach to evaluate the expected return of changing clinical guidelines. It also presents robust evaluation schemas that assist the multivariate data-oriented approach in selecting the best model. The application is focused on patients with ATTRv Amyloidosis. To present and characterize the work done, this thesis is structured into four main sections. It begins with an introduction and a presentation of ATTRv Amyloidosis from a medically historic perspective. Then it presents the relevant background by dwelling into the connection of time to event prediction with feature engineering, model construction and model evaluation, as well as introducing key concepts of genealogical studies. After this, it presents its technical contributions, in the form of the main publications that constitute this work (one paper by chapter). It ends with an epilogue section which overviews the work performed, shares the main conclusions, and, finally, discusses the thesis from a technical and clinical perspective.

Keywords: Time-to-Event Data; Data Modeling; Regression Models