Titulo Estágio
MOODoke App: Emotion-based Karaoke App
Áreas de especialidade
Engenharia de Software
Sistemas Inteligentes
Local do Estágio
DEI
Enquadramento
Given its major social importance, particularly in the digital society, music plays an important role in the world economy. In 2017, digital music revenues rose to US$7.8 billion, accounting for 50% of the global music revenues, while the global recorded music market grew by 5.9%.
It is expected that the amount of digital music continues to explode. Digital music repositories need, then, more advanced, flexible and user-friendly search mechanisms, adapted to the requirements of individual users. This has led to an increased awareness to the Music Information Retrieval (MIR) area. Several companies, e.g., Google, Pandora, Spotify, Apple, Sony or Philips, have set up MIR research agendas, with commercial applications already in place.
Within MIR, Music Emotion Recognition (MER) emerged as a significant sub-field. In fact, “music’s preeminent functions are social and psychological”, and so “the most useful retrieval indexes are those that facilitate searching in conformity with such social and psychological functions. Typically, such indexes will focus on stylistic, mood, and similarity information”.
Besides the music industry, the range of applications of MER is wide and varied, e.g., game development, cinema, advertising or health (e.g., music therapy).
MER research promises significant social, economic and cultural repercussions, as well as substantial scientific impact. There are presently several open and complex research problems in this multidisciplinary field, touching fields such as audio signal processing, natural language processing (NLP), feature engineering and machine learning.
In this context, as part of a cooperation between the MIR group at the Center of Informatics and Systems of the University of Coimbra (CISUC) and the Laboratory of Music and Computational Audio, Academia Sinica, Taiwan, we developed a computational prototype entitled MOODoke: Emotion-based Karaoke, which can be found at http://moodoke.dei.uc.pt/. The prototype is based on music emotion recognition in the context of karaoke, resorting to YouTube videos. From these videos, song lyrics are automatically extracted through optical character recognition (OCR – this module is already developed), from which lyrics-based music emotion variation detection (MEVD) is to be performed.
Objetivo
So far, only the application front-end and the OCR module have been developed. As such, the objectives to be achieved in this project are the following:
1. To update the current system via
a. The integration of the emotion-based song verses classification module
b. The integration of the audio-based classification module
c. Code cleaning and optimization, to make it publicly available, possibly including code re-factoring and/or migration of components developed in Matlab
2. To develop a corresponding standalone app
3. To evaluate the complete solution, both in terms of software quality requirements (e.g., computational performance) and classification accuracy
Plano de Trabalhos - Semestre 1
1. Familiarization with the application and code (1 month; Sep/2020)
2. State of the art review (1 month; Oct/2020)
3. Integration of the current emotion-based audio and song verses classification module (2 months; Nov-Dec/2020)
4. Writing of the Intermediate Report
Plano de Trabalhos - Semestre 2
5. System optimization and refactoring (3 months; Jan-Mar/2021)
6. Development of the standalone app (2 months; Abr-Mai/2021)
7. Complete system evaluation (1 month; May/2021)
8. Writing of a scientific paper and thesis (1 month; June/2021)
Condições
Possibility of scholarship in the second semester (6 months, 835.98 EUR/month). At this moment, this is a likely possibility, but that still needs to be confirmed.
Orientador
Rui Pedro Paiva, Prof. Renato Panda, Prof. Ricardo Malheiro
ruipedro@dei.uc.pt 📩