How to utilize D455 sensor to colorize 3d point cloud map created using lidar

[dinossht]

Is there an easy way to colorize the point cloud map created using icp odometry and slam? I have a D455 realsense depth camera and a Livox mid360 lidar. SLAM is performed using mid360 and I want to test colorizing the point cloud.

Hope anyone can help!

[matlabbe]

If you feed the lidar point cloud and the camera RGB image to rtabmap, it is possible to colorize the lidar point cloud in post-processing.

For example, after opening the database in standalone RTAB-Map or rtabmap-databaseViewer, you can do File->Export clouds... and choose this config:



Some examples:

 * http://official-rtab-map-forum.206.s1.nabble.com/Livox-Avia-Depth-cameras-and-organized-point-clouds-tp9200p9220.html

 * slightly different approach that could color online the lidar point cloud: http://official-rtab-map-forum.206.s1.nabble.com/Multi-monocular-cameras-tp9383p9396.html

[dinossht]

Thank you Mathieu!

If I were to opt for applying color offline, what is the best way to add the color images to the database? Currently I run pure icp odometry and odometry slam for mapping. And add rgb topic to the SLAM node?


[UPDATE]
I got it seemingly to work by adding rgb and camera_info topics, and setting subscribe_rgb=true. Also adding approx_sync=true

[matlabbe]

dinossht wrote

[UPDATE]
I got it seemingly to work by adding rgb and camera_info topics, and setting subscribe_rgb=true. Also adding approx_sync=true

Yes, that what I was referring. You may also try with odom_sensor_sync to adjust position of the camera based on icp odometry. For best results, I also recommend to calibrate extrinsics between the camera and the lidar. CAD values can be a good start to set the transform between the sensors, but could worth it to try if calibration would produce a better registration of the color on the point cloud. For packages doing so, you may start looking here.

cheers,
Mathieu

[dinossht]

Great tips, thanks Mathieu!

I will look into these.

[dinossht]

And is it as easy to incorporate 360 RGB camera images? Is it possible to add distorted images? Are there any challenges with 360 cameras, or is it as straight forward as "regular" FOV cameras?

[matlabbe]

 I guess you are referring to cameras like ricoh or insta360. With current code no. Would it be possible if we add support to 360 calibration format, probably. I am not sure what kind of calibration (omni, fisheye, pinhole?) or image formats these cameras can provide. Following links of this post (ricoh and OpenVSLAM integration) could give an idea what kind of data we can get.

[dinossht]

Thank you, and I will look into it.
I'll share the findings if I get to integrate 360 images into the rtabmap.