Re: Combine 3D Lidar and RGB-D Camera to create a single 3D point cloud (sensor fusion)

Posted by Alex Frantz on
URL: http://official-rtab-map-forum.206.s1.nabble.com/Combine-3D-Lidar-and-RGB-D-Camera-to-create-a-single-3D-point-cloud-sensor-fusion-tp10162p10482.html

Hi Mathieu,

Thanks a lot for your input and for your time :)

I was able to re-create your proposed solutions successfully, although they do not directly solve my problem, they do allow me to start thinking about other ways to reach my goal. I will be considering the option of involving an external node to filter out the Lidar points with the RGB ones created by the map_assembler.

While experimenting with all this, I came across a different approach. I decided to try the following:

- Before starting RTAB-MAP. I tried to first colorize the lidar point cloud using the RGB information from the camera. Basically, project the lidar points into the RGB image, extract the color, and assign it to the points from the Lidar point cloud that are currently in the FoV of the camera. An example of what I am describing can be found in this repo: https://github.com/leo-drive/color-point-cloud.

- Once the lidar point cloud is colorized, I tried launching RTAB-MAP by providing the colorized point cloud topic to the rtabmap node (/scan_cloud remaped to colorized point cloud topic). I also stopped using the rgbd_sync node since I don´t need the RGB data anymore, assuming that RTABMAP can extract the RGB fields from the colorized point cloud directly and build the map incrementally. Grid/Sensor is set to 0 to use the Lidar for the occupancy grid.

Unfortunately, the /cloud_map topic is still giving a segmented map. That is, ground and obstacles color coded green/red respectively. Looks like RGB information assigned to the points after the colorization step is lost, and replaced with values representing ground or obstacle.  

I was wondering if my approach makes sense, or if RTABMAP is currently capable of adapting to my approach. If not, I will go ahead and try your other proposal (involving an external node).

Thanks and looking forward to your reply!

Cheers,

Alex