3D object detection is able to obtain the position, size and orientation information of objects in 3D space, and is widely used in automatic driving perception, robot manipulation, and other applications. In 3D object detection, sensors such as LiDAR, RGB camera and depth camera are commonly used. In recent years, several works have been proposed to utilize 4D radar as a primary or secondary sensor to achieve 3D object detection. 4D radar, also known as 4D millimeter wave (mmWave) radar or 4D imaging radar. Compared to 3D radar, 4D radar not only obtains the distance, direction and relative velocity (Doppler velocity) of the target object, but also detects the height of the object. Due to its robustness against different weather conditions and lower cost, 4D radar is expected to replace low beam LiDAR in the future. This post summarizes the 4D radar based 3D object detection methods and datasets and will be continuously updated at https://github.com/liuzengyun/Awesome-3D-Detection-with-4D-Radar.
Basic Knowledge
Different 4D Radar Data Representations
- PC: Point Cloud
- ADC: Analog-to-Digital Converter signal
- RT: Radar Tensor
Datasets
| Dataset | Sensors | Radar Data | Source | Annotations | url | Other |
|---|---|---|---|---|---|---|
| Astyx | 4D Radar,LiDAR, Camera | PC | 19'EuRAD | 3D bbox | github paper | ~500 frames |
| RADIal | 4D Radar,LiDAR, Camera | PC, ADC, RT | 22'CVPR | 2D bbox, seg | github paper | 8,252 labeled frames |
| View-of-Delft(VoD) | 4D Radar,LiDAR, Stereo Camera | PC | 22'RA-L | 3D bbox | website | 8,693 frames |
| TJ4DRadSet | 4D Radar,LiDAR, Camera, GNSS | PC | 22'ITSC | 3D bbox, TrackID | github paper | 7,757 frames |
| K-Radar | 4D Radar,LiDAR, Stereo Camera, RTK-GPS | RT | 22'NeurIPS | 3D bbox, TrackID | github paper | 35K frames; 360° Camera |
| Dual Radar | dual 4D Radars,LiDAR, Camera | PC | 23'arXiv | 3D bbox, TrackID | paper | 10K frames |
| L-RadSet | 4D Radar,LiDAR, 3 Cameras | PC | 24'TIV | 3D bbox, TrackID | github paper | 11.2K frames; Annos range to 220m |
SOTA Papers
From 4D Radar Point Cloud
- RPFA-Net: a 4D RaDAR Pillar Feature Attention Network for 3D Object Detection (21'ITSC)
- Multi-class road user detection with 3+1D radar in the View-of-Delft dataset (22'RA-L)
- 🔗Link: paper
- 🏫Affiliation:
- 📁Dataset: VoD
- 📖Note: baseline of VoD
- SMURF: Spatial multi-representation fusion for 3D object detection with 4D imaging radar (23'TIV)
- 🔗Link: paper
- 🏫Affiliation: Beihang University (Bing Zhu)
- 📁Dataset: VoD, TJ4DRadSet
- 📖Note:
- PillarDAN: Pillar-based Dual Attention Attention Network for 3D Object Detection with 4D RaDAR (23'ITSC)
- 🔗Link: paper
- 🏫Affiliation: Shanghai Jiao Tong University (Lin Yang)
- 📁Dataset: Astyx
- 📖Note:
- MVFAN: Multi-view Feature Assisted Network for 4D Radar Object Detection (23'ICONIP)
- 🔗Link: paper
- 🏫Affiliation: Nanyang Technological University
- 📁Dataset: Astyx, VoD
- 📖Note:
- SMIFormer: Learning Spatial Feature Representation for 3D Object Detection from 4D Imaging Radar via Multi-View Interactive Transformers (23'Sensors)
- 🔗Link: paper
- 🏫Affiliation: Tongji University
- 📁Dataset: VoD
- 📖Note:
- RadarPillars: Efficient Object Detection from 4D Radar Point Clouds (24'arXiv)
- 🔗Link: paper
- 🏫Affiliation: Mannheim University of Applied Sciences, Germany
- 📁Dataset: VoD
- 📖Note:
Fusion of 4D Radar & LiDAR
- InterFusion: Interaction-based 4D Radar and LiDAR Fusion for 3D Object Detection (22'IROS)
- 🔗Link: paper
- 🏫Affiliation: Tsinghua University (Li Wang)
- 📁Dataset: Astyx
- 📖Note:
- Multi-Modal and Multi-Scale Fusion 3D Object Detection of 4D Radar and LiDAR for Autonomous Driving (23'TVT)
- 🔗Link: paper
- 🏫Affiliation: Tsinghua University (Li Wang)
- 📁Dataset: Astyx
- 📖Note:
- L4DR: LiDAR-4DRadar Fusion for Weather-Robust 3D Object Detection (24'arXiv)
- 🔗Link: paper
- 🏫Affiliation: Xiamen University
- 📁Dataset: VoD, K-Radar
- 📖Note: For the K-Radar dataset, we preprocess the 4D radar spar setensor by selecting only the top 10240 points with high power measurement. This paper is submitted to 25'AAAI.
- Robust 3D Object Detection from LiDAR-Radar Point Clouds via Cross-Modal Feature Augmentation (24'ICRA)
Fusion of 4D Radar & RGB Camera
- LXL: LiDAR Excluded Lean 3D Object DetectionWith 4D Imaging Radar and Camera Fusion (24'TIV)
- 🔗Link: paper
- 🏫Affiliation: Beihang University (Bing Zhu)
- 📁Dataset: VoD, TJ4DRadSet
- 📖Note:
- RCFusion: Fusing 4-D Radar and Camera With Bird’s-Eye View Features for 3-D Object Detection (23'TIM)
- 🔗Link: paper
- 🏫Affiliation: Tongji University
- 📁Dataset: VoD, TJ4DRadSet
- 📖Note:
Others
- Towards Robust 3D Object Detection with LiDAR and 4D Radar Fusion in Various Weather Conditions (24'CVPR)
Representative researchers
- Li Wang (Postdoctoral Fellow) and his co-leader Xinyu Zhang @Tsinghua University
- Bing Zhu @Beihang University
- Lin Yang @Shanghai Jiao Tong University
- Chris Xiaoxuan Lu @University College London (UCL)