Support for Multi-Camera IR Stereo Mode in RTAB-Map

[GiladB]

Hi Mathieu,

I'm working on a project involving multiple RealSense D455 cameras and exploring the possibility of operating them in IR stereo mode (left + right IR images) instead of the standard RGB+D mode.

I wanted to ask:

1. Does RTAB-Map currently support using multiple cameras in stereo mode using only their IR left and right images, without relying on the onboard depth or RGB streams?

2. Do you think this mode could offer any advantages over the D455's RGB+D mode, such as improved accuracy or better performance in harsh lighting conditions?

3. If this is not currently supported, and assuming it could be beneficial, what would be required to enable such a mode in RTAB-Map?

Best regards,
Gilad

[matlabbe]

1. Yes, use stereo_sync to sync images of each camera, and use stereo_odometry. Make sure to to build rtabmap with OpenGV to support multi-cameras. You can also connect the sensor_data output topic of stereo_odometry to rtabmap node so that the exact synchronized images are also used in slam node. See also this post. Ideally, if you can hardware sync the shutter between the cameras it will improve the result.

2. Maybe for outdoor when features are very far away, for which no depth would be computed by the realsense camera (though there is maybe a setting to get super dense depth). If you were using a D435, I would say that doing IR+Depth mode is better for perfect depth sync with left IR camera, and that the IR cameras have less motion blur and larger FOV. For D455, I think all 3 cameras have similar spec and RGB would be synchronized with the IR cameras.

From its datasheet, the D455:

• Intel® RealSense™ Vision Processor D4
• Up to 1280 x 720 stereo depth resolution
• Up to 1280 x 800 RGB resolution
• Diagonal field of view over 90°
• Dual global shutter sensors for up to 90 FPS depth
streaming
• RGB global shutter sensor for up to 90 FPS
• Range 0.4 m to over 6 m (varies with lighting
conditions)
• Inertial Measurement Unit (IMU) for 6 Degrees of
Freedom (6DoF) data
• D455f camera has IR-pass filter
• D456 is similar to D455 with an IP65 enclosure

Section 3.7.4
The external sensor connector provides the interface for external sensors to synchronize to depth output.
For D415, D455, D455f and D456 both the depth and RGB sensors can be synced by an external trigger.
For D435, D435i, D435f and D435if, only the depth sensor can be synced by an external trigger.
For D401, the external sync mechanism is not exposed.

I am not sure if "both the depth and RGB sensors can be synced by an external trigger" also means that color and depth are already synced internally (I think so if they do externally). Note that this something important that the image used for visual features is perfectly synced with the depth image (which is the case when we use left IR camera as the reference camera).

cheers,
Mathieu