Titulo Estágio
Concept Drift and Online Learning for Online Failure Prediction
Áreas de especialidade
Sistemas Inteligentes
Engenharia de Software
Local do Estágio
CISUC-SSE
Enquadramento
Several techniques have been developed over the years to avoid or handle faults (e.g., testing, coding practices). In particular, Online Failure Prediction (OFP) attempts to predict the occurrence of failures in the near future by combining past data and the current system state [Salfner, F., Lenk, M., & Malek, M. (2010). A survey of online failure prediction methods. ACM Computing Surveys (CSUR), 42(3), 1-42.]. Such predictions allow taking preemptive measures to avoid, or at least mitigate, their consequences.
Despite its potential OFP is still not widely adopted. One of the reasons is that failures are rare, and thus failure data (to create the predictive models) is typically not available. To overcome this issue recent work has used fault injection to generate realistic failure data.
Given the recent technological developments, ML has been successfully used in a variety of complex problems. In short, ML algorithms are able to identify and learn from complex patterns in the data often without relying on a predetermined model and use them to make predictions on new unseen data. Using failure data generated through fault injection recent work has successfully used ML to create accurate failure predictors.
One of the limitations of the recent related literature is that they all consider stationary environments, that is, they assume the workload of the system will always be the same. While this may be true for some cases, many modern problems are dynamic and change over time. This may lead to changes in the data distribution, which is known as concept drift [Schlimmer, J. C., & Granger, R. H. (1986). Incremental learning from noisy data. Machine learning, 1(3), 317-354]. If the business process changes it is likely that so will the failures that the system will experience (as different workloads exercise different subsystems). Ultimately, existing failure predictors will no longer accurately model the problem and their performance will start to drop. Various approaches have been proposed in the ML research field to deal with this issue, such as incremental and online learning [Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4), 1-37].
The focus of this internship is on exploring concept drift and incremental/online learning in the context of OFP. The goal is to study how changes in the business process affect the performance of failure predictors, how to detect such changes in the data, and how to update the predictive models to mitigate the effects of concept drift. Fault injection techniques will be used to generate data and simulate concept drift.
Objetivo
The learning objectives of this master internship are:
1) Dependability, fault tolerance, fault injection: study the subject of fault tolerance, focusing on fault injection, as means to improve the dependability of modern systems
2) Online Failure Prediction: understand the problem of OFP and how it can be used to predict and mitigate incoming failures
3) “Classical” Machine Learning: understand how to use and the impact of classical ML techniques on creating accurate models
4) “Advanced” Machine Learning: study more advanced ML concepts such as concept drift and batch/incremental/online learning to deal with realistic non-stationary environments
5) Research Design: understand how to design and execute an experimental process to address complex and open research issues
Plano de Trabalhos - Semestre 1
[20/09/2021 a 04/10/2021] Literature review:
Study the concepts to be used in the internship, namely online failure prediction, “classical” machine learning, concept drift, and online learning
[05/10/2021 a 08/11/2021] Analysis and selection of target techniques:
Identification, analysis, and selection of which concept drift and online learning techniques will be studied
[09/11/2021 a 30/11/2021] Definition of the experimental process:
Design and plan the experimental process that will be used to conduct the study. This includes defining all the relevant components, from simulating and detecting concept drift to updating the predictors, as well as the architecture of the testbed that will be used
[01/12/2021 a 15/01/2022] Write the dissertation plan
Plano de Trabalhos - Semestre 2
[07/02/2022 a 6/03/2022] Set up the experimental testbed:
Set up the testbed required to conduct the experiments. This comprises several tasks, such as simulating and detecting concept drift, to implementing mechanisms to allow online learning.
[7/03/2022 a 17/04/2022] Conduct the experimental campaign:
Use the testbed to conduct the experimental process, considering different types and degrees of concept drift as well as different ML methods
[18/04/2022 a 08/05/2022] Explore and assess the generated data:
Process, explore, and analyze the generated data to understand the behavior of the predictors under concept drift and the effectiveness of the different techniques considered
[09/05/2022 a 04/06/2022] Write a technical report.
[17/05/2022 a 30/06/2022] Write the thesis.
Condições
Depending on the evolution of the internship a studentship may be available to support the development of the work. Also, the work is to be executed at the laboratories of the CISUC’s Software and Systems Engineering Group. A workplace will be provided as well as the required computational resources.
Observações
Trabalho a ser co-orientado por Marco Vieira
Orientador
João R. Campos
jrcampos@dei.uc.pt 📩