Titulo Estágio
Using Netflix Conductor to Orchestrate Microservices
Áreas de especialidade
Engenharia de Software
Local do Estágio
DEI
Enquadramento
The leading consulting company McKinsey estimates that there will be a shortage of data scientists to enable organizations to explore the full potential of big data. By 2018, the United States alone will face a shortage of 140,000 to 190,000 professionals with strong analytical skills with the know-how to analyze big data to make effective decisions. This shortage will be more dramatic in Portugal since, in contrast to US universities that provide Data Science degrees for several years (e.g., at Berkeley and Carnegie
Mellon University), Portuguese universities are just making the first steps.
This shortage of professionals cannot be mitigated easily, since training students to become data scientists requires time and resources to teach skills from diverse knowledge areas such as Computer Science, Statistics, Business, and Data Visualization.
Hence, the objective of the FCT DataScience4NP project is to explore the use of visual programming paradigms to enable non-programmers to be part of the Data Science workforce. More specifically, the objective of the DataScience4NP project is to build Cloud Native Applications (CNA) for Data Science using microservices.
Objetivo
This thesis will develop one of the components of the DataScience4NP platform. Namely, the main goal is to customize the Netflix orchestration engine called Conductor to manage and execute complex workflows driving data science applications.
Data Science applications can be described using processes (workflows), which consist of multiple distinct interconnected analytical microservices provided as-a-service. Data scientists can describe these processes and upload them to an orchestrator (workflow engine), which takes care of state management, correct execution order, parallelism, and synchronization. The use of an orchestrator facilitates the control and visualization of the interactions between the microservices. To enable processes to be reusable and adaptable, workflow templates and variability modeling are key ingredients to explore.
Specific Objectives:
• To customize Netflix Conductor orchestration engine
• To develop workflow templates for microservices orchestration
• To integrate Conductor with API gateway service
Technologies:
• http://techblog.netflix.com/2016/12/netflix-conductor-microservices.html
Plano de Trabalhos - Semestre 1
- Review of the state of the art and technologies on workflow templates and variability modeling
- Requirement analysis (including both functional and non-function requirements)
- System architecture
- Writing of the preliminary thesis
Plano de Trabalhos - Semestre 2
- System development
- System testing
- Writing of the final thesis
Condições
The student might receive a scholarship from the FCT DataScience4NP project (745€ / month).
Orientador
1) Jorge Cardoso, 2) Rui Pedro Paiva, 3) Filipe Araújo
ruipedro@dei.uc.pt 📩