Re: Combine 3D Lidar and RGB-D Camera to create a single 3D point cloud (sensor fusion)
Posted by Alex Frantz on
URL: http://official-rtab-map-forum.206.s1.nabble.com/Combine-3D-Lidar-and-RGB-D-Camera-to-create-a-single-3D-point-cloud-sensor-fusion-tp10162p10461.html
Hi again,
Thanks a lot for the input, indeed the transforms between the camera and LiDAR were not accurate. I remounted the sensors on the robot, checked with the CAD model and redefined the transforms in the URDF. The point clouds are now aligned much better, yet not perfect. As you suggested, camera LiDAR calibration is required which will be one of my next steps! :D
That said, I am facing a new issue/concern and would like your input on this if possible please. In this thread, we previously discussed how to combine both point clouds (generated from RGB-D and LiDAR) to generate a single colored point cloud. I transferred this into a simulation to get perfect results for testing purposes.
I discovered however that I am not getting the desired result. Perhaps I did not previously explain what I needed well, so I will try to be as detailed and concise as possible. Sorry for the added confusion..
My goal is to use RTABMAP to generate a single point cloud, which is colored incrementally by the RGB-D camera online. That is, as I am moving my robot inside an environment, I want the points that have already been scanned by the LiDAR to be colored based on the RGB data from the camera. If there are points that are only visible from the LiDAR, I want them to also remain in the generated point cloud, and updated with color if the camera sees them. I know that is possible to do in post-processing via the database viewer and exporting the colored LiDAR point cloud, but I would like this done online instead.
I did some research and stumbled upon these threads in the forum:
1. http://official-rtab-map-forum.206.s1.nabble.com/Livox-Avia-Depth-cameras-and-organized-point-clouds-td9200.html#a9220 | Possible online coloring of the LiDAR point cloud
2. http://official-rtab-map-forum.206.s1.nabble.com/How-to-utilize-D455-sensor-to-colorize-3d-point-cloud-map-created-using-lidar-td10438.html | Coloring the point cloud in post-processing
3. http://official-rtab-map-forum.206.s1.nabble.com/Painting-a-pointcloud-td8130.html | Looks like this person was able to color the point cloud online but I cannot reproduce this
Based on these findings, I updated the launch file in an attempt to achieve the desired results but I have not been successful.
What I noticed is that while the initial scan of the environment seems to properly color the LiDAR point cloud with the RGB information, the moment the camera is looking somewhere else, the colored points are immediately replaced by the LiDAR scan, that is, they lose their color. If the camera passes again from a previously scanned location, the points without color (I assume generated from the LiDAR) are replaced with colored ones (I guess generated from the RGB-D camera), and once the camera looks somewhere else, the colored ones are replaced once again with non-color values.
Here is a visual example of what I tried explaining so far:
Notice how in the left image, the points in the FoV of the camera are colored, but once the robot rotates to the right (right-hand image) and the camera FoV changes, these previously colored points are then overwritten by the LiDAR scan values. Similarly, the previously scanned points from LiDAR scans, are replaced with colored ones generated from the camera. I suspect that the nodes in charge of mapping are overwriting themselves in the current setup.
To generate this results I am using the following RTABMAP launch file rtabmap_livox_rgbd_lidar.py.
I saw some examples were they set the following parameters: RGBD/CreateOccupancyGrid : False and Grid/Sensor : 1 (default), and instead add:
'gen_depth': True,
'gen_depth_decimation': 8,
'gen_depth_fill_holes_size':1,
But this simply results in a uncolored point cloud generated only from LiDAR in RVIZ. See below:
The only way that I can somehow incorporate color information in the LiDAR point cloud via the RGB-D camera, is if I set :
'RGBD/CreateOccupancyGrid': 'true',
'Grid/Sensor': '2',
I know this has been quite a long post, but I tried including as much information as possible. To summarize, my question is whether it is possible to configure RTABMAP to color the LiDAR point cloud online using RGB-D data, and display this in RVIZ by subscribing to the /cloud_map topic. The whole goal is to use the 3D LiDAR to map the entire environment and complement it with RGB-D data once the camera sees an already scanned area online.
The repository I am using for the simulation environment is the following: https://github.com/Alexandre-Frantz/leo_simulator-ros2
Also, perhaps this is a long shot but I would be also open to schedule an online meeting with you to perhaps better explain or show you live the results I am getting. I understand however if this is not possible, but at this point I am going around in circles trying to figure this out.
I am happy to also provide you with more code should you need it for testing reasons!
Thanks a lot for your time and for all your help so far, I am looking forward to your inputs!
Best,
Alex
URL: http://official-rtab-map-forum.206.s1.nabble.com/Combine-3D-Lidar-and-RGB-D-Camera-to-create-a-single-3D-point-cloud-sensor-fusion-tp10162p10461.html
Hi again,
Thanks a lot for the input, indeed the transforms between the camera and LiDAR were not accurate. I remounted the sensors on the robot, checked with the CAD model and redefined the transforms in the URDF. The point clouds are now aligned much better, yet not perfect. As you suggested, camera LiDAR calibration is required which will be one of my next steps! :D
That said, I am facing a new issue/concern and would like your input on this if possible please. In this thread, we previously discussed how to combine both point clouds (generated from RGB-D and LiDAR) to generate a single colored point cloud. I transferred this into a simulation to get perfect results for testing purposes.
I discovered however that I am not getting the desired result. Perhaps I did not previously explain what I needed well, so I will try to be as detailed and concise as possible. Sorry for the added confusion..
My goal is to use RTABMAP to generate a single point cloud, which is colored incrementally by the RGB-D camera online. That is, as I am moving my robot inside an environment, I want the points that have already been scanned by the LiDAR to be colored based on the RGB data from the camera. If there are points that are only visible from the LiDAR, I want them to also remain in the generated point cloud, and updated with color if the camera sees them. I know that is possible to do in post-processing via the database viewer and exporting the colored LiDAR point cloud, but I would like this done online instead.
I did some research and stumbled upon these threads in the forum:
1. http://official-rtab-map-forum.206.s1.nabble.com/Livox-Avia-Depth-cameras-and-organized-point-clouds-td9200.html#a9220 | Possible online coloring of the LiDAR point cloud
2. http://official-rtab-map-forum.206.s1.nabble.com/How-to-utilize-D455-sensor-to-colorize-3d-point-cloud-map-created-using-lidar-td10438.html | Coloring the point cloud in post-processing
3. http://official-rtab-map-forum.206.s1.nabble.com/Painting-a-pointcloud-td8130.html | Looks like this person was able to color the point cloud online but I cannot reproduce this
Based on these findings, I updated the launch file in an attempt to achieve the desired results but I have not been successful.
What I noticed is that while the initial scan of the environment seems to properly color the LiDAR point cloud with the RGB information, the moment the camera is looking somewhere else, the colored points are immediately replaced by the LiDAR scan, that is, they lose their color. If the camera passes again from a previously scanned location, the points without color (I assume generated from the LiDAR) are replaced with colored ones (I guess generated from the RGB-D camera), and once the camera looks somewhere else, the colored ones are replaced once again with non-color values.
Here is a visual example of what I tried explaining so far:
Notice how in the left image, the points in the FoV of the camera are colored, but once the robot rotates to the right (right-hand image) and the camera FoV changes, these previously colored points are then overwritten by the LiDAR scan values. Similarly, the previously scanned points from LiDAR scans, are replaced with colored ones generated from the camera. I suspect that the nodes in charge of mapping are overwriting themselves in the current setup.
To generate this results I am using the following RTABMAP launch file rtabmap_livox_rgbd_lidar.py.
I saw some examples were they set the following parameters: RGBD/CreateOccupancyGrid : False and Grid/Sensor : 1 (default), and instead add:
'gen_depth': True,
'gen_depth_decimation': 8,
'gen_depth_fill_holes_size':1,
But this simply results in a uncolored point cloud generated only from LiDAR in RVIZ. See below:
The only way that I can somehow incorporate color information in the LiDAR point cloud via the RGB-D camera, is if I set :
'RGBD/CreateOccupancyGrid': 'true',
'Grid/Sensor': '2',
I know this has been quite a long post, but I tried including as much information as possible. To summarize, my question is whether it is possible to configure RTABMAP to color the LiDAR point cloud online using RGB-D data, and display this in RVIZ by subscribing to the /cloud_map topic. The whole goal is to use the 3D LiDAR to map the entire environment and complement it with RGB-D data once the camera sees an already scanned area online.
The repository I am using for the simulation environment is the following: https://github.com/Alexandre-Frantz/leo_simulator-ros2
Also, perhaps this is a long shot but I would be also open to schedule an online meeting with you to perhaps better explain or show you live the results I am getting. I understand however if this is not possible, but at this point I am going around in circles trying to figure this out.
I am happy to also provide you with more code should you need it for testing reasons!
Thanks a lot for your time and for all your help so far, I am looking forward to your inputs!
Best,
Alex
| Free forum by Nabble | Edit this page |