Titulo Estágio
Body motion and face expression classification under real time motion-tracking conditions in virtual reality environments
Área Tecnológica
Reconhecimento de Padrões
Local do Estágio
IBILI/FMUC - Instituto Biomédico de Investigação em Luz e Imagem / Faculdade de Medicina Universidade de Coimbra
Enquadramento
The human-computer interaction (HCI) paradigms evolved drastically in the past few years. With the emergence of Natural User Interfaces (NUIs) such as Speech, Touch and Gesture, the ways we have to interact with machines become everyday one step closer to the ways we have to interact with each others.
With those advances, several applications emerge in the areas of human behaviour. At IBILI's Virtual Reality Lab, we aim to leverage such mechanisms to create virtual interactive experiences that mimic the social interactions we have in real life, to help in the study of patients with social disorders, like autism.
In 2010, Microsoft® released Kinect, a low-cost 3D camera with capabilities of real-time skeleton tracking that won the Guinness World Record for "Fastest Selling Consumer Electronics Device". This award is a good indicator of people's acceptance of these new technologies. In our VR lab we also have a state-of-the-art professional full body motion capture system (OptiTrack® Arena Bundle), which will be used in this study along with Microsoft® Kinect.
Objetivo
The main objective of this project is to create algorithms to identify and classify specific motions of social interactions, aiming its application in rehabilitation scenarios in virtual reality environments. Two types of movements will be targeted:
1 - Body movements, based on skeleton positions extracted from Microsoft® Kinect or body markers detected through OptiTrack® Arena Bundle.
The student is asked to create an algorithm to identify movements like: greeting, hugging, handshaking, etc. This includes the recording of an annotated dataset for the training and validation of the algorithms.
2 - Facial expressions, based on facial markers detected through OptiTrack® Expression.
It is expected the creation of an algorithm with capabilities of identification of facial expression like: happy, sad, angry, etc. This includes the recording of an annotated dataset for the training and validation of the algorithms.
3 - Integration with Virtual Reality environment.
Application of the algorithm in the Virtual Reality system. Only simple proof-of-concept scenarios are expected.
Plano de Trabalhos - Semestre 1
17 Sep - 31 Oct
State of the art review - Review of the most recent approaches to this problem in the state of the art. Identification of the biggest challenges in the field.
1 Nov - 31 Nov
Approach - Definition of the requirements and how will the problem be approached. Definition of the protocols for the creation of body motion and facial expression datasets. Definition of how to integrate the algorithms in the Virtual Reality environments.
1 Dez - 31 Dez
Body motion dataset creation - Recording sessions and posterior data analysis and motion annotation. Several aspects must be achieved in the dataset, such as diversity and coverage.
1 Jan - 28 Jan
1st semester report - Writing and reviewing of the first semester report.
Plano de Trabalhos - Semestre 2
15 Feb - 8 Mar
Facial expression dataset creation - Recording sessions and posterior data analysis and facial expressions annotation.
9 Mar - 31 Mar
First algorithm implementation - Implementation and application of the first implementation. Results analysis and definition of possible improvements.
1 Apr - 30 Apr
Algorithm improvements - Final implementation and application. Results analysis and comments.
1 May - 31 May
Virtual Reality system integration - Integration of the algorithm with the virtual reality system.
1 Jun - 28 Jun
Dissertation - Writing and reviewing of the dissertation document.
Condições
The workplan will be performed in IBILI-FMUC, where the student will be given a workstation and full support of a heterogeneous team composed by Informatics Engineers, Biomedical Engineers and Psychologists, with experience both on the Motion Capture and Virtual Reality systems.
Our Virtual Reality lab is full equiped with state-of-the-art hardware which can be fully used by the student.
Payment:
A research grant (BI) can be considered depending on the curriculum vitae and motivation of the applicant.
Observações
Experience on pattern recognition problems will be preferred.
Orientador
Marco Simões
msimoes@dei.uc.pt 📩