Titulo Estágio
Evaluation of test suite generation using AI
Áreas de especialidade
Engenharia de Software
Local do Estágio
CISUC
Enquadramento
Generative AI is poised to transform many aspects of software engineering, with one of the most promising applications being the automatic generation of unit test suites. Given the critical role of testing in identifying and preventing software defects (bugs), leveraging AI to streamline and enhance test creation has significant potential.
However, despite their impressive capabilities, advanced AI models—particularly large language models (LLMs)—are known to produce hallucinations or inaccurate outputs. This represents a risk in the context of software testing, as it may result in test cases that are incorrect or ineffective. Even more concerning, such flawed test suites could fail to detect critical bugs, undermining the reliability of the software they are meant to safeguard.
Objetivo
The objective of this thesis is to evaluate the ability of test suites generated by AI models to detect software bugs across different codebases. In addition to evaluating performance, this thesis aims to obtain insights into how AI models generate test suites (e.g., based on the source code, based on the parameters, randomly), the types of bugs that tend to escape detection, and whether different prompts or configurations can generate better test suites. The evaluation will focus on open-source and free models, such as Code LLaMa, DeepSeek-Coder v2.0, StarCoder, Phi-3, Mistral-7B, CodeT5, Polycoder, etc. These models will be used to produce a dataset of test suites, which will then be evaluated using mutation testing and coverage metrics.
Plano de Trabalhos - Semestre 1
T1 (M1-M3) - Review of the state-of-the-art in generative AI/LLMs, AI-generated test suites and mutation testing
T2 (M4) - Collection of AI models to be used in experiments
T3 (M4) - Collection of diverse open-source repositories to be used in experiments
T4 (M5) - Writing the intermediate report
Plano de Trabalhos - Semestre 2
T5 (M6-M7) - Generation of test suites
T6 (M8-M10) - Evaluation of generated test suites using mutation testing and coverage metrics, followed by comparison with human-generated test suites
T7 (M10-M11) - Writing the final report
Condições
The dissertation will take place in the research centre CISUC (SSE group).
Depending on the results, there is the possibility of writing a research paper to be submitted to a top scientific conference or journal.
Orientador
Frederico Cerveira
fmduarte@dei.uc.pt 📩