Titulo Estágio
Programmable Vibrotactile Sensations for Virtual Reality
Áreas de especialidade
Sistemas de Informação
Engenharia de Software
Local do Estágio
DEI
Enquadramento
Recent advances in neuroscience have led to the development of clinical rehabilitation protocols for several different medical problems, ranging from Alzheimer to severe physical impairment, that makes use of interactive, immersive virtual environments to train and (or) induce distinctive signal patterns in the patients’ neuroactivity[2]. Given that the effectiveness of the use of VR interactive systems in clinical treatments is deeply dependent on the system’s realism and sensible feedback[3], tactile feedback devices are much required. In this context, the rehabilitation protocol devised to implement a brain-machine interface that enables an impaired paraplegic patient to control an Exo-skeleton robot device using EEG signals [4], uses tactile feedback of the VR patients avatar legs, delivered to the patients arms, through vibrating actuators that can perform distinct intensity and frequency patterns, in order to stimulate specific neurological activity that in long term induce the patients neuroplasticity.
This project proposal is closely related to the above mentioned example, and the research results and products will be used in the development of the tactile feedback devices that will be used in the Thertact-Exo project (Premio Melo e Castro, Santa Casa Neurociências 2018).
In another perspective, the VR systems and applications that are used in medical research uses the same development tools and methodologies used in the implementation of general multimedia applications such as games and interactive experiences. In this regard, the development of a dedicated software package for controlling generic tactile feedback hardware that can be used through high level programming and embedded into already existent VR development frameworks such as Unity and A-Frame is an important contribution that will simplify and accelerate the VR application development from design to test to deployment.
Objetivo
The objectives of this project are to:
- Develop a vibration pattern editor: a software tool to specify vibration patterns. The tool should be flexible enough to allow different number and configuration of actuators, allow users to visualize the vibration pattern over time, to test and calibrate the vibration patterns on the hardware, and to export the pattern into a machine- and human-readable format.
- Develop Virtual Reality components for A-Frame and Unity to control the vibration actuators according to a pre-configured vibration pattern, or dynamically (i.e., the software should allow developers to “play” a vibration pattern exported with the previous tool, or programatically create a pattern on-the-fly. These components must be developed according to the best-practices of each VR framework.
- Develop demonstration Virtual Reality applications that showcase the tool and components.
- Assess through subject trials the tactile discrimination regarding vibration intensity, localization and pattern recognition.
Developing the tool and VR components may require developing additional control software to interface with the hardware that controls the vibration actuators (Arduino).
The developed software should be well structured and follow software design patterns and object-oriented best practices.
The developed VR components should be usability-tested, i.e., their programmatic interface should be evaluated with other VR programmers to understand how their interfaces can be improved.
Plano de Trabalhos - Semestre 1
1. Study haptic sensations, particularly vibrotactile sensations and how they have been applied in the medical area, particularly in VR applications.
2. Study the existing hardware and understand in detail how it works (type of actuators, how they are controlled through Arduino)
3. Study the target frameworks, namely Unity and A-Frame, particularly regarding how to program external components that require communication with custom hardware
4. Create visual prototypes for the editor, define its architecture and implement a limited-functionality prototype.
Plano de Trabalhos - Semestre 2
5. Implement the editor, perform a usability and discrimination evaluation, and use the collected data to improve the hardware/software performance.
6. Implement the components for A-Frame and Unity, perform a usability evaluation with programmers, and improve them.
7. Implement demo VR applications that showcase the flexibility of the editor and components.
8. Write a scientific paper, and dissertation.
Condições
A hardware prototype of the vibrotactile device with be available during the course of the project.
Observações
Trabalho em co-orientação com o prof. Doutor André Perrotta do Centro de Investigação em Ciência e Tecnologia das Artes, no Porto.
Referências:
[1] http://hapticdesign.github.io/macaron/#
[2] Lebedev, M. A., & Nicolelis, M. A. (2006). Brain–machine interfaces: past, present and future. TRENDS in Neurosciences, 29(9), 536-546.
[3] Slater, M., Khanna, P., Mortensen, J., & Yu, I. (2009). Visual realism enhances realistic response in an immersive virtual environment. IEEE computer graphics and applications, 29(3), 76-84.
[4] Donati, A. R., Shokur, S., Morya, E., Campos, D. S., Moioli, R. C., Gitti, C. M., ... & Brasil, F. L. (2016). Long-term training with a brain-machine interface-based gait protocol induces partial neurological recovery in paraplegic patients. Scientific reports, 6, 30383.
Orientador
Jorge C. S. Cardoso
jorgecardoso@dei.uc.pt 📩