FreeZe: Training-Free Zero-Shot 6D Pose Estimation with Geometric and Vision Foundation Models

Estimating the 6D pose of objects unseen during training is highly desirable yet challenging. Zero-shot object 6D pose estimation methods address this challenge by leveraging additional task-specific supervision provided by large-scale, photo-realistic synthetic datasets. However, their performance heavily depends on the quality and diversity of rendered data and they require extensive training. In this work, we show how to tackle the same task but without training on task-specific data. We propose FreeZe, a novel solution that harnesses the capabilities of pre-trained geometric and vision foundation models. FreeZe generates distinctive 3D point-level descriptors by combining 3D geometric descriptors learned from unrelated 3D point cloud datasets and 2D visual features learned from web-scale 2D image collections. We then estimate the 6D pose of unseen objects via RANSAC-based 3D registration. We also introduce a novel algorithm based on visual features to resolve ambiguities caused by geometrically symmetric objects. We comprehensively evaluate FreeZe across the seven core datasets of the BOP Benchmark, which include over a hundred 3D objects and 20,000 images captured in various scenarios. FreeZe consistently outperforms all state-of-the-art approaches, including competitors extensively trained on synthetic 6D pose estimation data. Code is publicly available at andreacaraffa.github.io/freeze.