Titulo Estágio
Computational statistics tools for the experimental evaluation of stochastic multiobjective optimisers
Área Tecnológica
Sistemas Evol. e Comp.
Local do Estágio
CISUC-DEI
Enquadramento
The experimental evaluation of the performance of stochastic multiobjective optimisers is currently of great interest, especially to the evolutionary multiobjective optimisation (EMO) research community. A typical scenario involves the application of a number of alternative optimisers to an appropriate set of problem instances, and the statistical analysis of the run-time and/or solution-quality results thus obtained.
Due to their stochastic nature, evolutionary multiobjective optimisers typically produce a different set of non-dominated solutions in each run, according to some (unknown) probability distribution, which depends both on the problem instance and on the optimiser itself. Ultimately, the experimental evaluation of the performance of stochastic multiobjective optimisers consists of estimating and comparing aspects of the corresponding set distributions, which may also include a run-time dimension.
In the literature, two main approaches to the characterisation of the performance of EMO algorithms have been adopted. The performance-indicator approach consists of defining scalar indicators of the quality of non-dominated sets produced in each run, such as the hypervolume indicator, leading to a simpler, univariate statistical analysis. The attainment-function approach, on the other hand, directly addresses the non-dominated set distribution in objective space. In any case, when it comes to comparing the performance of several optimisers, suitable hypothesis tests are needed to allow statistically sound decisions to be made.
Hypothesis tests based on the empirical attainment function (EAF) have been previously formulated. However, they can be rather computationally demanding, and the lack of available tools has limited their adoption.
Objetivo
This work is concerned mainly with the development of computational tools for the evaluation of stochastic multiobjective optimisers in common experimental settings such as the selection of the best from a number of optimizers, or the comparison of newly developed algorithms with a state-of-the-art algorithm (the control). In practice, such experimental designs typically involve multiple influence factors, blocking, and multiple testing issues, which are not covered by current EAF-based testing procedures, for example.
Integration of such tools with existing numerical and scientific visualisation environments such as python, numpy and matplotlib or mayavi in order to make them easily available to the practitioner is another goal of this work.
Plano de Trabalhos - Semestre 1
1. Literature review on evolutionary multiobjective optimisers and performance assessment.
2. Familiarisation with the theoretical concepts of statistical experimental design, permutation tests and multiple comparisons, including closed test procedures.
3. Implementation of EAF-based permutation tests for the two-sample case on different hardware platforms (ideally CPUs and GPUs).
4. Preliminary evaluation of the algorithms developed.
Plano de Trabalhos - Semestre 2
5. Description of the experimental set-ups of interest and specification of the statistical test problems to be considered.
6. Implementation of the corresponding statistical hypothesis tests.
7. Comparative study involving multiple EMO algorithms and multiple optimisation problems.
8. Interpretation and discussion of the results.
9. Dissertation writing.
Condições
The ideal candidate will have a strong background in at least two of the following areas: statistics, high-performance computing and evolutionary computation. The work will be carried out in the CISUC ECOS laboratories. A computer cluster is available to support the development and evaluation of the algorithms developed.
Orientador
Carlos M Fonseca e Viviane Grunert da Fonseca
cmfonsec@dei.uc.pt 📩