Candidate
Tiago André Queiroz Soares da Costa
Date, Time and Location:
September 16, 14:30, Sala de Atos da Faculdade de Engenharia da Universidade do Porto
President of the Jury:
Doutor Carlos Miguel Ferraz Baquero-Moreno, Professor Catedrático da Faculdade de Engenharia da Universidade do Porto
Members:
Klara Nahrstedt, PhD, Full Professor, Department of Computer Science, University of Illinois at Urbana-Champaign, United States of America;
Pedro António Amado de Assunção, PhD, Coordinator Professor, Departamento de Engenharia Eletrotécnica, Escola Superior de Tecnologia e Gestão do Instituto Politécnico de Leiria;
Luís António Pereira de Meneses Corte-Real, PhD, Associate Professor, Departamento de Engenharia Eletrotécnica e de Computadores, Faculdade de Engenharia da Universidade do Porto;
Maria Teresa Magalhães da Silva Pinto de Andrade, PhD, Assistant Professor, Departamento de Engenharia Eletrotécnica e de Computadores, Faculdade de Engenharia da Universidade do Porto (Supervisor).
Abstract:
This thesis proposes a novel approach to immersive, multiview media distribution that uses Deep Learning models and user-centric data to predict user interest in the near future while multimediacontent is being presented. The main objective of this thesis is to give the user a truly ubiquitousmultimedia immersive experience without the need for expensive equipment, while also allowing him or her to see the scene being presented on the screen from almost any angle, as if they were actually there when the scene was shot. A methodological approach was envisioned based on the literature review and the identification of gaps in immersive streaming scenarios, which resulted in the conceptualization of a brand new architecture that was coined Smooth Multiview (SmoothMV). This architecture is capable of analysing user behaviour data in real-time and preemptively preparing content delivery accordingly based on viewing interests demonstrated by users while visualising a particular scene. Users effortlessly provide behaviour data without equiring intrusive equipment, which is then processed using the novel concept of the Hot&Cold matrix, which this thesis describes. With the use of this concept, the screen is divided into nine separate regions, each of which is connected to a neighbouring view that the SmoothMV architecture can prepare and choose to present. Separate queues designated for playback and buffering of upcoming content segments are introduced to provide minimal delay without compromising the user experience, since content adaptation is closely linked to user inputs. The number of views that are available and the approach employed for analysing behaviour while viewing content and selecting which view should be processed in the following moment affect how these queues are managed. This thesis developed from a purely reactive approach to a sophisticated, twofold Deep Learning architecture that can accurately identify the view that best fits the interests of the user with a high degree of accuracy. The development of a new dataset was needed in order to achieve this level of performance, as the data provided by existing datasets was not suitable for the scenario that was proposed. After a series of 128 experiments were conducted to collect visual attention data from 45 participants while viewing multi-perspective content, the Data2MV dataset was created and made available to the public. This thesis’ fundamental concepts and practical outputs are considered to be of significant importance to the body of knowledge currently available in the field of research, while also offering relevant tools for the general enhancement of current content distribution architectures.
Keywords: Multimedia, Streaming, Multiview, Focus-of-Attention, Deep Learning