Titulo Estágio
Implicit fisheye depth estimation
Áreas de especialidade
Sistemas Inteligentes
Local do Estágio
ISR-Polo 2
Enquadramento
Fisheye cameras have a wider field of view than ordinary pinhole cameras. This comes at the cost of
heavy distortions. However, a lot of methods depend on images without this distortion: structure
from motion, image stitching, stereo-vision, photogrammetry, …
Alternatively, we can calibrate a system of fisheye cameras (two or more) implicitly. This means we
do not calibrate the individual cameras, but the system as a whole. We already did this for non-
fisheye cameras here: https://medium.com/@xandervermeulen/exploring-multi-camera-3d-object-
tracking-using-neural-networks-1b7848b3dfc1. In a nutshell, we can learn the mapping from 2D
pixels for all cameras combined to 3D world coordinates using AI. Afterwards, when presented with
a new point visible in all the cameras, we can use the AI to predict the 3D coordinate.
Objetivo
Calibrating a camera means finding values for all its parameters, including the distortion ones. Most
methods assume some sort of geometrical model for fisheye cameras, which often is an
oversimplification of the underlying reality. Recent work overcomes this, by turning a fisheye camera
in a pinhole model:
De Boi, I., Pathak, S., Oliveira, M., Penne, R. (2024). How to Turn Your Camera into a Perfect Pinhole
Model. In: Vasconcelos, V., Domingues, I., Paredes, S. (eds) Progress in Pattern Recognition, Image
Analysis, Computer Vision, and Applications. CIARP 2023. Lecture Notes in Computer Science, vol
14469. Springer, Cham. https://doi.org/10.1007/978-3-031-49018-7_7
The core if this idea is to use machine learning to capture the distortion and thus being able to
remove it. Once the distortion is removed, we can use it to stitch images together or do depth
estimation.
Alternatively, we can calibrate a system of fisheye cameras (two or more) implicitly. This means we
do not calibrate the individual cameras, but the system as a whole. We already did this for non-
fisheye cameras here: https://medium.com/@xandervermeulen/exploring-multi-camera-3d-object-
tracking-using-neural-networks-1b7848b3dfc1. In a nutshell, we can learn the mapping from 2D
pixels for all cameras combined to 3D world coordinates using AI. Afterwards, when presented with
a new point visible in all the cameras, we can use the AI to predict the 3D coordinate.
Plano de Trabalhos - Semestre 1
The goal of this project is to implement the implicit camera calibration method and compare it to
the state-of-the-art.
Objectives:
1. A Matlab or Python codebase to calibrate a system of fisheye cameras implicitly.
2. Use this system to perform depth estimation.
Plano de Trabalhos - Semestre 2
3. Compare it to the state-of-the-art, such as for instance the articles:
R. Komatsu, H. Fujii, Y. Tamura, A. Yamashita and H. Asama, "360° Depth Estimation from Multiple
Fisheye Images with Origami Crown Representation of Icosahedron," 2020 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA, 2020, pp. 10092-10099,
doi: 10.1109/IROS45743.2020.9340981. Example in Figure 3.
Condições
Estágio nos laboratórios do ISR-UC.
Orientador
Helder Araújo
helder@isr.uc.pt 📩