Candidate:
Adriana Silva Souza
Date, time and place:
July 10, 10:30, Sala de Atos FEUP
President of the Jury:
António Fernando Vasconcelos Cunha Castro Coelho, PhD, Associate Professor with Habilitation, Faculdade de Engenharia da Universidade do Porto.
Members:
Vitor Manuel Pereira Duarte dos Santos, PhD, Assistant Professor, NOVA Information Management School, Universidade Nova de Lisboa;
João Manuel Pereira Barroso, PhD, Associate Professor with Habilitation, Vice-Reitor para a Inovação, Transferência de Tecnologia e Universidade Digital, Universidade de Trás-os-Montes e Alto Douro;
João Paulo Ramos Teixeira, PhD, Coordinator Professor, Departamento de Eletrotecnia, Escola Superior de Tecnologia e Gestão do Instituto Politécnico de Bragança;
Maria Selene Henriques da Graça Vicente, PhD, Assistant Professor, Departamento de Psicologia, Faculdade de Psicologia e de Ciências da Educação da Universidade do Porto;
Maria do Rosário Marques Fernandes Teixeira de Pinho, PhD, Associate Professor, Departamento de Engenharia Eletrotécnica e de Computadores, Faculdade de Engenharia da Universidade do Porto;
Diamantino Rui da Silva Freitas, PhD, Associate Professor, Departamento de Engenharia Eletrotécnica e de Computadores, Faculdade de Engenharia da Universidade do Porto (supervisor).
Abstract:
“The synthesized speech of mathematical contents still presents several challenges. For Mathematics to be understood by people with visual impairment, it needs to be verbalized in detail, which generates long outputs and causes cognitive overload; in addition, Mathematics has quite peculiar rules. Therefore, most of the time, prosodic limits such as pauses, and intonation are not adequately synthesized. This investigation proposes a model that uses prosody and interaction to access mathematical expressions to minimize the problems mentioned. We relied on the Design-Based Research methodology to develop the model and divided the study into four stages. In the first stage, a systematic literature review was carried out. We conducted an initial exploration investigation with interviews with students with visual impairments and braille teachers and analyzed the mathematics spoken by speech synthesizers. In the second stage of the investigation, a corpus of mathematical expressions spoken by math professors was created to support prosody research. Intonation and pauses were the investigated prosodic components. Although the studies have yet to go into intonation in-depth, we did some tests of prosodic modulation of the fundamental frequency, highlighting stretches of mathematical expressions according to the level in the MathML tree. Concerning pauses, we identified their main patterns in mathematical expressions. We also performed an eye-tracking experiment with sighted people to understand the cognitive processes surrounding mathematical expressions’ observation, analysis, and processing. In the third stage, a linear regression model which calculates the pauses for mathematical expressions dynamically was created and evaluated with visually impaired students. The results showed advances regarding the solutions found, perceived mainly when the mathematical expressions are unfamiliar to the students. The results of the eye tracking experiment showed that in addition to the complexity of the mathematical expression, it was necessary to propose a new formal concept that was called diversity, quantifying this subjective property of the structures of expressions because it was found that it also impacts during the cognitive processing of expressions. Data analysis provided clues for creating the interaction model that uses diversity to control the cognitive load in accessing mathematical expressions during the process. The evaluation of the model with visually impaired people showed an advance concerning existing works since students performed better when accessing mathematical expressions with the model. In the fourth stage, we made the final proposition of the model based on the assessment of people with visual impairments. The results achieved in this investigation allow greater autonomy in the reading of mathematical expressions; people with visual impairment can govern the interaction in the auditory access according to the need to reinforce their memory; in addition, it can reduce the time in the manipulation of mathematical expressions compared to traditional tools, improve the writing process, since reading is linked to writing and relieve the student’s memory. In addition to these contributions, we can also highlight the discovery of the new diversity parameter, which is strongly related to the cognitive processing of expressions. In general, these contributions make it possible to improve and develop mathematics education, particularly in the teaching-learning process of visually impaired people, making them more autonomous beings, which, in addition to scientific contributions, can also generate social and economic impacts arising from accessibility.”
Keywords: Synthesized Speech, Mathematics, Accessibility, Visual Impairment, Complexity, Diversity.