Titulo Estágio
Co-design and Development of an Interactive Sensory Substitution Prototype
Local do Estágio
DEI-FCTUC
Enquadramento
Individuals with visual disabilities are often excluded from the richness of everyday experiences that rely on visual features [1]. This creates an opportunity for technology to go beyond information delivery to support novel, more accessible ways of interacting with those experiences, from natural landscapes to visual art.
Sensory Substitution Devices (SSDs) emerged in the 1960s, when Paul Bach-y-Rita introduced the idea of using tactile displays to “see with the skin” through devices such as the Tactile Vision Substitution System [2]. These experiments demonstrated the brain’s plasticity and the potential of cross-modal perception [3,4]. However, despite decades of development, SSDs have yet to fulfill the promises of fully replacing lost senses, particularly vision [5,6].
Additionally, many SSDs fail to consider the emotional or hedonic aspects of perception, reducing the user experience to spatial or functional information [7]. Most systems also lack adaptability, do not align with natural human sensory experience, and fail to support real-time interaction in complex environments [8,9,10].
One domain where this need is particularly urgent is in the experience of visual art, which remains largely inaccessible to audiences with visual disabilities [11]. Museums are increasingly exploring methods, such as sonification, to provide aesthetic and emotional access to visual artworks for visitors with visual impairments [12].
These studies reinforce the need for systems that go beyond information delivery and open new possibilities for designing auditory stimuli capable of recreating the emotional experience associated with visual perception.
Objetivo
This dissertation aims to co-design, develop, and evaluate an accessible and interactive prototype of a sensory substitution device. The system will translate specific visual images (such as paintings) into emotionally expressive soundscapes. The project includes co-design sessions with visually impaired individuals to gather requirements, the development of the system, and further co-design sessions to assess emotional perception and interpretability.
Plano de Trabalhos - Semestre 1
Conduct a literature review on sensory substitution devices (SSDs), human-centered design, and human perception of visual aesthetics.
Review of recommendations and design guidelines for user interfaces for visually impaired people.
Define user requirements based on videos and testimonials from individuals with visual impairments, along with the prior reviews.
Validate and adjust the requirements for the sensory substitution prototype through a co-design session with participants with visual impairments.
Write an intermediate dissertation document.
Plano de Trabalhos - Semestre 2
Iterative design and development of an accessible and interactive prototype of a sensory substitution device that translates visual input into emotionally expressive auditory feedback.
Evaluate the system’s accessibility using heuristic evaluation methods with established guidelines.
Evaluate the prototype’s emotional effectiveness and interpretability through co-design sessions with end-users.
Review the intermediate dissertation document (if needed) and write the remaining chapters of the dissertation.
Condições
The student is expected to have an interest in Human-Computer Interaction and Accessible Computing, as well as good implementation skills. Fluency in English is a plus.
Observações
The work will be co-supervised by Pedro Garruço, PhD student in Computational Media Design, who researches the topic of this dissertation.
References
[1] Tobias, E. I., & Mukhopadhyay, S. (2017). Disability and Social Exclusion: Experiences of Individuals with Visual Impairments in the Oshikoto and Oshana Regions of Namibia. Psychology and Developing Societies, 29(1), 22–43. doi:10.1177/0971333616689203
[2] Bach-Y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision Substitution by Tactile Image Projection. Nature, 221(5184), 963–964. doi:10.1038/221963a0
[3] Schmidmaier, M. (2011). Sensory Substitution Systems.
[4] Howarth, C. I. (1972). Review: Brain Mechanisms in Sensory Substitution. Perception, 1(4), 491–492. doi:10.1068/p010491
[5] Auvray, M., & Harris, L. R. (2014). The State of the Art of Sensory Substitution. Multisensory Research, 27(5–6), 265–269. doi:10.1163/22134808-00002464
[6] Lenay, C., Gapenne, O., Hanneton, S., Marque, C., & Genouëlle, C. (2003). Chapter 16. Sensory substitution: Limits and perspectives. In Y. Hatwell, A. Streri, & E. Gentaz (Eds.), Touching for Knowing: Cognitive psychology of haptic manual perception (pp. 275–292). doi:10.1075/aicr.53.22len
[7] Loomis, J. (2003). Sensory replacement and sensory substitution: Overview and prospects for the future.
[8] Maidenbaum, S. (2015, April). Practical Sensory Substitution In Real And Virtual Worlds: Development, Accessibility And Neuroscience. Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, 211–214. doi:10.1145/2702613.2702626
[9] Proulx, M. J., Gwinnutt, J., Dell’Erba, S., Levy-Tzedek, S., de Sousa, A. A., & Brown, D. J. (2015). Other ways of seeing: From behavior to neural mechanisms in the online “visual” control of action with sensory substitution. Restorative Neurology and Neuroscience, 34(1), 29–44. doi:10.3233/RNN-150541
[10] Huang, J. (2023, September). An overview of current prosthetic sensory rehabilitation and future development. International Conference on Modern Medicine and Global Health (ICMMGH 2023), 12789, 191–196. doi:10.1117/12.2692419
[11] Eardley, A. F., Thompson, Hannah, Fineman, Anna, Hutchinson, Rachel, Bywood, Lindsay, & Cock, M. (2022). Devisualizing the Museum: From Access to Inclusion. Journal of Museum Education, 47(2), 150–165. doi:10.1080/10598650.2022.2077067
[12] Dini, S., Ludovico, L. A., Mascetti, S., & Valero Gisbert, M. J. (2023, April). Translating Color: Sonification as a Method of Sensory Substitution within the Museum. Proceedings of the 20th International Web for All Conference, 162–163. doi:10.1145/3587281.3587706
Orientador
Paula Alexandra Silva, Pedro Garruço
paulasilva@dei.uc.pt 📩