Posted by
matlabbe on
URL: http://official-rtab-map-forum.206.s1.nabble.com/Further-Processing-of-camera-images-RTAB-data-best-way-to-do-it-tp3015p3025.html
Hi,
Thx referring
YOLO object detection, I didn't know about that, kinda cool.
1.
I assume you are on ROS? If so, the /rtabmap/mapData output can be used to get incremental data to map. For example, you can look at the
rtabmap_ros/MapCloud rviz plugin (which is based on
official PointCloud2 rviz plugin) to see how new data is added to 3D map.
In
MapData msg, there is the optimized graph of the map and only the latest node of "nodes" contains the latest data added to map. So only data of one node is published at after each map update (at default 1 Hz), keeping low bandwidth. Like rtabmapviz or rtabmap_ros/MapCloud rviz plugin, the cloud of the new data is created and added to their list of clouds kept in cache on their side. The optimized graph is used to adjust the poses of all created clouds so far if a loop closure happened for example.
2.
If the timestamps of YOLO results are set accordingly to image captured, TF can be used to get a pose in the past. Note the TF has limited history buffered, I think the time to keep past transforms can be increased by a parameter. With the graph sent by rtabmap, we could find the node having the same timestamp than the detection (if same image is used for rtabmap than YOLO) or get the two nodes with timestamps before and after the YOLO result. The pose can be then interpolated between the two nodes, similar to this
example where the WiFi captures were done between nodes.
3.
Is the goal getting the object 3D pose referred to map? You would need to keep the depth image along with RGB image used with YOLO, to be able to triangulate its 3D local pose in camera frame. Then to get in map frame, the approach above could be used to get 3D pose accordingly to map's graph. You can also look at the
find_object_2d's 3D example to see TF of objects published after a detection.
4.
The graph published in /rtabmap/mapData contains always all poses in the map, optimized if there was a loop closure. To detect if the poses have changed, either compare all poses with some poses cached from previous msg or use
/rtabmap/info topic (which can be exactly synchronized with mapData) and see if "loopClosureId' and "proximityDetectionId" are non-zero.
cheers,
Mathieu