Open-Set Semantic Ray Frontiers for Online Scene Understanding and Exploration

Omar Alama, Avigyan Bhattacharya, Haoyang He, Seungchan Kim,
Yuheng Qiu, Wenshan Wang, Cherie Ho, Nikhil Keetha, Sebastian Scherer
Carnegie Mellon University
Paper arXiv Video Code

RayFronts is a real-time semantic mapping system that enables fine-grained scene understanding both within and beyond the depth perception range allowing robots to localize and effectively limit search volumes.

RayFronts can be queried with open-set images and text within the map and beyond it.

Abstract

Open-set semantic mapping is crucial for open-world robots. Current mapping approaches either are limited by the depth range or only map beyond-range entities in constrained settings, where overall they fail to combine within-range and beyond-range observations. Furthermore, these methods make a trade-off between fine-grained semantics and efficiency. We introduce RayFronts, a unified representation that enables both dense and beyond-range efficient semantic mapping. RayFronts encodes task-agnostic open-set semantics to both in-range voxels and beyond-range rays encoded at map boundaries, empowering the robot to reduce search volumes significantly and make informed decisions both within & beyond sensory range, while running at 8.84 Hz on an Orin AGX. Benchmarking the within-range semantics shows that RayFronts's fine-grained image encoding provides 1.34x zero-shot 3D semantic segmentation performance while improving throughput by 16.5x. Traditionally, online mapping performance is entangled with other system components, complicating evaluation. We propose a planner-agnostic evaluation framework that captures the utility for online beyond-range search and exploration, and show RayFronts reduces search volume 2.2x more efficiently than the closest online baselines.

Video

System Design

Overview of our online mapping system, RayFronts is designed for multi-objective & multi-modal open-set querying of both in-range and beyond-range semantic entities. Given posed RGB-D images, we first extract dense features with our fast language-aligned image encoder. Then, posed depth information and features are used to construct a semantic voxel map for in-range queries. In parallel, RayFronts maintains a VDB-based occupancy map to generate frontiers, which are further associated with multi-directional semantic rays. The semantic ray fronts enables beyond-range querying of open-set concepts in the unmapped region.

Online Semantic Mapping

Our proposed planner-agnostic online-mapping evaluation metric captures how online mapping systems correctly cuts down search volume.

RayFronts is superior to all baselines across depth ranges empowering both fine-grained localization and beyond-range guidance.

Offline 3D Open-Vocabulary Semantic Segmentation

RayFronts consistently outper-forms the baselines in mIoU, and achieves SOTA performance beating the next best baselines by +18.07% and +9.63% mIoU on Replica and Scannet, respectively, excluding background. RayFronts is also able to handle background seamlessly with its single-forward pass approach while segment-then-encode approaches fall short. For outdoor in-the-wild performance on TartanAirV2, RayFronts exceeds the performance of the baselines by 3.36% mIoU

Encoder & Mapping Throughput Analysis

RayFronts provides state-of-the-art mIoU & 17.5 Hz throughput on an AGX Orin. It surpasses Trident with 1.34x higher mIoU and a 16.5x speedup, while achieving 1.81x higher mIoU than NACLIP, which operates at a similar throughput. We also evaluate the full mapping throughput and achieve real time performance at 8.84 Hz on the AGX Orin.

How to use in your own project ?

  • 🤖 Guide your robot with semantics within and beyond. RayFronts can be easily deployed as part of your robotics stack as it supports ROS2 inputs for mapping and querying and has robust visualizations.
  • 🖼️ Use our SOTA encoder. If you are only interested in our open-vocabulary semantic segmentation encoder, then the RayFronts encoder is also available as a standalone file independent of the rest of the codebase.
  • 🚀 Bootstrap your semantic mapping project. Utilize the modular and clean RayFronts mapping codebase with its supported datasets to build your project.
  • 💬 Reach out or raise an issue if you face any problems !

BibTeX

@misc{alama2025rayfrontsopensetsemanticray,
      title={RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Exploration}, 
      author={Omar Alama and Avigyan Bhattacharya and Haoyang He and Seungchan Kim and Yuheng Qiu and Wenshan Wang and Cherie Ho and Nikhil Keetha and Sebastian Scherer},
      year={2025},
      eprint={2504.06994},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2504.06994}, 
}