penIns3D: Snap and Lookup for
3D Open-vocabulary Instance Segmentation

ECCV 2024

1University of Cambridge   2The University of Hong Kong   3The Hong Kong University of Science and Technology (Guangzhou) 4The Hong Kong University of Science and Technology

Demo of OpenIns3D

OpenIns3D works well with complex input queries.

Abstract

In this work, we introduce OpenIns3D, a new framework for 3D open-vocabulary scene understanding that requires no aligned images as input. 🌟 The OpenIns3D framework employs a "Mask-Snap-Lookup" (🎭-πŸ“·-πŸ”) scheme. The "Mask" module 🎭 learns class-agnostic mask proposals in 3D point clouds, The "Snap" module πŸ“· generates synthetic scene-level images at multiple scales and leverages 2D vision-language models to extract interesting objects, and the "Lookup" module πŸ” searches through the outcomes of β€œSnap'' to assign category names to the proposed masks. This approach, yet simple, achieves state-of-the-art performance across a wide range of 3D open-vocabulary tasks, including recognition, object detection, and instance segmentation, on both indoor and outdoor datasets. Moreover, OpenIns3D facilitates effortless switching between different 2D detectors without requiring retraining. When integrated with powerful 2D open-world models, it achieves excellent results in scene understanding tasks. Furthermore, when combined with LLM-powered 2D models, OpenIns3D exhibits an impressive capability to comprehend and process highly complex text queries that demand intricate reasoning and real-world knowledge.

General Pipeline

Your Image Description

OpenIns3D is a unique pipeline for 3D open world scene understanding without 2D input

Your Image Description
OpenIns3D Features

Open-vocabulary/Zero-shot Instance Segmentation Results Demo on on S3DIS

Input Queries: 'ceiling', 'floor', 'wall', 'beam', 'column', 'window', 'door', 'table', 'chair', 'sofa', 'bookcase', 'board'

Open-vocabulary/Zero-shot Instance Segmentation Results Demo on ScanNetV2

Input Queries: 'cabinet', 'bed', 'chair', 'sofa', 'table', 'door', 'window', 'bookshelf', 'picture', 'counter', 'desk', 'curtain', 'refrigerator', 'shower curtain', 'toilet', 'sink', 'bathtub'

Open-vocabulary/Zero-shot Instance Segmentation Results Demo on STPLS3D

Input Queries: 'building', 'vegetation', 'vehicle', 'truck', 'Aircraft', 'military vehicle', 'bike', 'motorcycle', 'light pole', 'street sign', 'clutter', 'fence'

BibTeX

@article{huang2023openins3d,
      title={OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation}, 
      author={Zhening Huang and Xiaoyang Wu and Xi Chen and Hengshuang Zhao and Lei Zhu and Joan Lasenby},
      journal={arXiv preprint},
      year={2023}
    }

Questions

Any question regarding the paper, please email Zhening via zh340 at cam.ac.uk