Titulo Estágio
Deep Structural Analysis: Automatic Detection and Segmentation of Song Sections for Lyrics Music Emotion Recognition
Áreas de especialidade
Sistemas Inteligentes
Local do Estágio
DEI-FCTUC
Enquadramento
The area of Music Emotion Recognition (MER), a sub-area of Music Information Retrieval (MIR), has received increasing interest from the international scientific community, both academic and industrial. In fact, companies such as Spotify, Pandora, Google, Sony, Philips Research, and Gracenote, among many others, have invested heavily in the area in recent years and used this aspect in their applications.
The recognition of music in emotions contemplates the recognition of emotions in the two dimensions of music (audio and song lyrics). This is because the emotional perception that a listener has of a song can depend on any of these two dimensions with variable weights, always depending on ultimately the listener and the music.
The classic process of building prediction models for MER begins by extracting features from the audio and lyrics separately. For example, in audio, we have features related to the detection of rhythm, melody, tempo, etc., and for lyric features, we have semantic, structural, stylistic, and content-based features. Then, the process continues with the construction of predictive models that use these features.
It is urgent to identify features that are more closely related to the domain of song lyrics, namely, the structure of song lyrics. In addition to sometimes being a little subjective, the identification of structures such as the chorus or bridge, the process of automatically extracting these structures is also not linear.
The objective of this project is to automatically detect and segment these structures based on song lyrics and then extract and evaluate features from them. The process involves natural language processing techniques, text mining, feature engineering, and applied machine learning knowledge.
Objetivo
The objectives to be achieved in this project are the following:
1. Research and development of musical structures commonly associated with music in general that can make sense in the Lyrics-based MER (LMER) process.
2. Research and development of NLP and feature engineering approaches associated with LMER, particularly those related to structural features.
3. Research and development of identification and segmentation of musical structures (e.g. chorus) in song lyrics.
Plano de Trabalhos - Semestre 1
1. State-of-the-art review: MER, NLP and feature engineering, machine learning for text in general and LMER in particular
2. Evaluation of the current algorithms (based on classical Natural Language Processing techniques and machine learning) on the novel datasets developed within the research group
3. Research and development of NLP and feature engineering approaches for static LMER (namely structural features, for the detection of segments such as chorus, etc.)
4. Writing of the Intermediate Report
Plano de Trabalhos - Semestre 2
5. Research and development of deep learning (DL) approaches for structural LMER analysis (e.g., via CNNs, transfer learning, etc.)
6. Critical evaluation of the developed DL models
7. Writing of a scientific paper and thesis
Condições
- Access to a toolbox for LMER (with feature extraction and deep learning modules), developed by our team
- Access to an LMER database, developed by our team
- Server access (hosted at DEI) with 10 high-performance GPU cards
Orientador
Rui Pedro Paiva, Ricardo Malheiro e Renato Panda
ruipedro@dei.uc.pt 📩