Titulo Estágio
A Visualization tool to Improve the Observability of Microservice Applications
Áreas de especialidade
Engenharia de Software
Local do Estágio
DEI
Enquadramento
## Summary
Breaking large software systems into smaller functionally interconnected components is a trend on the rise. This architectural style, known as “microservices”, simplifies development, deployment and management at the expense of complexity and observability. In fact, in large scale systems, it is particularly difficult to determine the set of microservices responsible for delaying a client’s request, when one module impacts several other microservices in a cascading effect. Components cannot be analyzed in isolation, and system operators lack an overall view of the system, to determine the bottlenecks and trace their root causes. To mitigate this problem, we propose to create a visualization tool capable of providing an overall view of the microservices system and enabling interaction with system administrators. This tool should advance the state of the art in monitoring, by providing a broad range of statistical and visual insights on the system.
## Scope
Distributed systems with fine-grained components are a growing trend in the industry. Microservice-based architectures and Function as a Service (FaaS) platforms are being favored for the flexibility they afford. This trend is only accentuated by the financial benefits and reduced development times promised by Platforms as a Service (PaaS) and serverless deployments. The benefits are faster development cycles, development team independence (because scope is limited and well defined), ease of deployment, management, scaling and governance. Microservices, and FaaS are the building blocks of modern, highly dynamic distributed systems. We can see this trend in many large international companies, like LinkedIn, Netflix, Uber, Ebay, Amazon, among many others, but the same trend can be found in important national companies as well.
On the downside, companies face increasing visibility challenges into the health and performance of their systems, as cloud and micro-service architectures become more complex and dynamic, with more changing parts. The complexity moves from the components to their interaction, and their emergent behaviors. Since developers and operators lack a complete view of the system, the result is impaired observability, which turns debugging and monitoring into a challenge that keeps getting harder, as systems grow larger with additional components and version iterations.
Tracing is one of the existing techniques for improving visibility in a distributed system. Informally it can be explained as tracking some causally related information through a system, e.g., as requests move from component to component, to enable reconstruction of the past state of the system. The idea is to log meaningful events in the system, like function calls, and send them to a central repository for later evaluation. This might allow, for example, the replay of complex interactions crossing multiple components of the system, to determine the cause for unusual delays. Unfortunately, distributed tracing systems, like Zipkin, lack good tools to perform a visual reconstruction and analysis of the (dynamic) architecture of the system and the existing interactions between microservices. This motivates us to create such tool.
This work should occur in the context of the DataScience for Non Programmers project (Data Science4NP - https://osf.io/8dc2e/).
Objetivo
The visualization tool should fetch data from a tracing system like Zipkin and provide an overall view of the existing services. It should allow system owners to inspect each one of these services, by zooming in and out from parts of the system, showing response distribution times of the services, enabling observation of the instances that comprise each service, provide hints regarding the importance of each interaction, among other features. In simple terms, the visualization tool should use tracing information from a distributed system of microservices, to enhance the observability of the system.
Plano de Trabalhos - Semestre 1
- Study the state of the art, namely existing tools, like Vizceral (https://github.com/Netflix/vizceral), and related technologies such as logging, metric collection, and tracing (1 month)
- Deploy the base components of the visualization tools (e.g., Kafka, Zipkin and Vizceral) (1 month)
- Define the requirements (1 month)
- Start the implementation (1 month)
- Write the intermediate report (1 month)
Plano de Trabalhos - Semestre 2
- Implement a basic version of the tool (3 months)
- Test the tool (1 month)
- Write final report (1 month)
Condições
This work should take place in the context of a research project funded by FCT. If the project gets an extension it may support a scholarship of 745 euros/month for 3 months with a possible extension.
Orientador
Filipe Araujo, Prof. Rui Paiva (Univ. of Coimbra) e Prof. Jorge Cardoso (Univ. of Coimbra/Huawei)
filipius@uc.pt 📩