misalignment from tango multi session

[sDeboffle]

Hi,
I tried to tild my lidar to avoid overlapping points (exemple on my last database, we can see the 8 layers of the lidar on some walls because my lidar was 0° tilt).
To avoid empty space on points cloud, I tried to tilt the lidar 30° forward. RTABmap is not able to map with the same ICP settings.
 I tried to tilt 10° forward, and that work, with less presicion.
Does the lidar need vertical wall detection on 360° of the lidar layers to make ICP registration?(with a 30° tilt, all 8 layers looks ground forward, ceiling backward and vertical wall on each side).

https://wetransfer.com/downloads/5dcdb8a07e751adcae2facf79890e2b320190510143738/83c2aa7b72e6d67a25f05e6e646aaab720190510143738/2ddfa3

cheers
stephane

[matlabbe]

Hi,

There could be have some ICP's parameters tuning to do for icp_odometry node (were you using icp_odometry node in this example?). If you can take a rosbag of the lidar data it could be easier for me to help you with the parameters.

cheers,
Mathieu

[sDeboffle]

Hi mathieu
I have made some other test and I'm able to have a good map with a 30° tilt lidar.
My launch file:
LidarImuKinect01.launch

all is good untill RTAB-map see a image similar to a previous image and make a loop closure. A have a yellow screen then RTAB map stop record points, and kinect image don't update anymore.

I have this message:
[rtabmap/rtabmap-4] process has died [pid 23910, exit code 127, cmd /home/sdeboffle/Bureau/Quanergy_launch/catkin_ws/devel/lib/rtabmap_ros/rtabmap -d scan_cloud:=assembled_cloud rgbd_image:=rgbd_image __name:=rtabmap __log:=/home/sdeboffle/.ros/log/4cd38968-8d1b-11e9-bf3f-9829a6388ce0/rtabmap-rtabmap-4.log].
log file: /home/sdeboffle/.ros/log/4cd38968-8d1b-11e9-bf3f-9829a6388ce0/rtabmap-rtabmap-4*.log

Can't find this file,sorry.

This is a repeatable issue. I have find that if I increase the graph optimization error from 1 to 3 for exemple, RTAB have less crash.
the DB:
https://wetransfer.com/downloads/2fb0db3b852feff9768be49267d5d59120190612152657/7ed1233bde08a2db73b872621e3b811a20190612152657/542417

Is my settings creates conflict, or is it possible to bypass this bug?
cheers
stephane

[matlabbe]

Hi Stephane,

If the bug is repeatable, you may try to start rtabmap node with gdb and "--udebug":

<node pkg="rtabmap_ros" type="rtabmap" name="rtabmap" output="screen" args="-d --udebug" launch-prefix="xterm -e gdb --args">

Then type "bt" wen it crashes to see the backtrace.

For the actual map, the density of the clouds added to rtabmap vary between 3000 to 320000 points. Is assembled_cloud always published at 1 Hz (if so you can safely set Rtabmap/DetectionRate to 0)? Does its size vary? For point_cloud_assembler, fixed_frame_id would need to be set to "odom" to actually assemble the clouds based on odometry displacement.

From the ouster example, the odometry local scan map size is set to 15000 instead of 1000, this would help to limit odometry drift:

<param name="OdomF2M/ScanMaxSize"        type="string" value="15000"/> 

Reg/Strategy could be set to 1 for rtabmap node.

cheers,
Mathieu

[sDeboffle]

Hi mathieu.
I modified my launch file:
LidarImuKinect01.launch
I have the same problem, but mapping looks even better than yesterday.

I tried with your command, for debug:
LidarImuKinect01-Debug.launch

The problem is that with this launch file, rtab-map don't display or record kinect data, so rtab-map doesn't crash.

I tried to play with a lot of setting to see if I can pass through the bug, but each time I have more than 40 images and a loop closure, RTAB-map freezing.

Do you have a other solution for me to sending you the usefull informations about this bug?

cheers,
stephane

[matlabbe]

sDeboffle wrote

The problem is that with this launch file, rtab-map don't display or record kinect data, so rtab-map doesn't crash.

subscribe_rgbd is true, so it should have images. For the bug, is it because you added gdb it is not crashing? Can you use the same launch file that is crashing and add gdb?

Also with "--udebug", you should have already more logs on the terminal, copy/paste here the last logs before rtabmap process dies.

cheers,
Mathieu

[sDeboffle]

Hi
I'vemade some test, but i'm not able to launch your command properly.
I got this message if I enter un preferences panel:
failed to get some RTAB-map parameters from ROS server, the rtabmap node may not started or some parameters won't work...
I've make 2 test,only udebug and only GDB
so...
for udebug, I got this:

[DEBUG] (2019-06-14 11:31:41.032) VWDictionary.cpp:930::addNewWords() total time 0.019120s
[DEBUG] (2019-06-14 11:31:41.032) VWDictionary.cpp:563::clear()
[DEBUG] (2019-06-14 11:31:41.034) VWDictionary.cpp:563::clear()
[DEBUG] (2019-06-14 11:31:41.035) RegistrationVis.cpp:1314::computeTransformationImpl()
[DEBUG] (2019-06-14 11:31:41.035) RegistrationVis.cpp:1327::computeTransformationImpl() words from3D=878 to2D=887
[DEBUG] (2019-06-14 11:31:41.036) util3d_motion_estimation.cpp:98::estimateMotion3DTo2D() words3A=878 words2B=887 matches=41 words3B=887 guess=xyz=0.000000,0.000000,0.000000 rpy=0.000000,-0.000000,0.000000
[DEBUG] (2019-06-14 11:31:41.041) RegistrationVis.cpp:1353::computeTransformationImpl() inliers: 29/41
[DEBUG] (2019-06-14 11:31:41.041) RegistrationVis.cpp:1462::computeTransformationImpl() Refine with bundle adjustment
[DEBUG] (2019-06-14 11:31:41.041) OptimizerG2O.cpp:1267::optimizeBA() Optimizing graph...
/home/sdeboffle/Bureau/Quanergy_launch/catkin_ws/devel/lib/rtabmap_ros/rtabmap: symbol lookup error: /usr/local/lib/librtabmap_core.so.0.19: undefined symbol: _ZN3g2o30OptimizationAlgorithmLevenbergC1EPNS_6SolverE
[ INFO] [1560504701.068338570]: Odom: ratio=0.746964, std dev=0.037634m|0.037634rad, update time=0.090078s
[ INFO] [1560504701.153869987]: Odom: ratio=0.722449, std dev=0.038325m|0.038325rad, update time=0.080172s
[rtabmap/rtabmap-4] process has died [pid 28324, exit code 127, cmd /home/sdeboffle/Bureau/Quanergy_launch/catkin_ws/devel/lib/rtabmap_ros/rtabmap -d --udebug scan_cloud:=assembled_cloud rgbd_image:=rgbd_image __name:=rtabmap __log:=/home/sdeboffle/.ros/log/12f11700-8e84-11e9-bf3f-9829a6388ce0/rtabmap-rtabmap-4.log].
log file: /home/sdeboffle/.ros/log/12f11700-8e84-11e9-bf3f-9829a6388ce0/rtabmap-rtabmap-4*.log


From GDB, I got this
Capture_du_2019-06-14_11-41-11.png


I'm able to pass through the bug by change the Vis/BundleAdjustment to 0 instead of 1.
I've some green and yellow screen during scan but no crash. So I can move around, make my scan, and reopen it to find loop closure with the database editor. Don't know if the result can be better in a other way but I can make my test like that and see if the settup is robust.

hope this can help
https://wetransfer.com/downloads/e6e19995214bb9064007d9bcccb6407020190614100347/f455b73f42c4debcdb61d3bde169827a20190614100347/7d61a5

https://wetransfer.com/downloads/c34adad9b1acfa61402630e010fad0b320190614152228/f0a80811b70aab4f9d6642b6d450bcc620190614152228/de7cb6

cheers
stephane

[matlabbe]

Hi,

It seems to be a g2o library version problem. Did you install also g2o from source (while the binaries are still installed)? The wrong g2o library (different than one used to build rtabmap) may be used on runtime.

You could log which libraries ldd is finding:

$ ldd /home/sdeboffle/Bureau/Quanergy_launch/catkin_ws/devel/lib/rtabmap_ros/rtabmap

In your case however, you can safely set Vis/BundleAdjustment to 0 as transforms are refined by lidar's scans.

In the first database, it happened two times that there was a very large point cloud saved in the database, like the voxel filtering done in Memory didn't work (Mem/LaserScanVoxelSize=0.1), causing a 10 sec lag before processing the next cloud. Here two consecutive clouds added to database, the yellow one is wrong, as it seems not downsampled. Not sure what caused the problem, a rosbag would be required to debug more.


For the second database, I can see that this configuration is difficult for scan matching in some situations where the lidar only sees the ceiling, ground and two parallel walls. Scan matching cannot know the lidar is moving in this case (it is missing constraints in x axis below), creating a drift.


If the lidar could be tilted less, to see more forward and behind, it could add those missing contraints. You could also decrease "Icp/PMOutlierRatio" to 0.3 or 0.4 for icp_odometry. For more debugging about odometry, a rosbag would be required (with a least lidar point cloud, IMU and TF between those sensors, images not required to debug odometry).

cheers,
Mathieu

[sDeboffle]

hi
I solve the g2o problem. I reinstall libg2o. that's work now.
I also change the lidar tilt from 30 to 10°.
Mapping is great but I'm limited from 600 to 1000 images per DB before RTABmap freeze mapping and console indicate :
[ WARN] [1560791253.260951213]: Could not get transform from Lidar to Lidar accordingly to odom after 0.100000 seconds (for stamps=1560791252.249110 -> 1560791252.942867)! Error="Lookup would require extrapolation into the future.  Requested time 1560791252.942867056 but the latest data is at time 1560791233.089997952, when looking up transform from frame [odom] to frame [Lidar]. canTransform returned after 0.100979 timeout was 0.1.".
how can I increase my scan duration without lossing in precision?
Is there a way to refine ICP in database viewer? or a way to access libpointmatcher settings on standelone?
If I stop my scan and restart a second scan at the same place, is this work like with tango multi-session?

https://wetransfer.com/downloads/05fa30d62d783c1fa228f2a48ff23b9420190618142130/7f3451ff12a2601fcc7ee8fb2f4ac6d620190618142130/87a148

cheers
stephane

[matlabbe]

Hi,

You are updating the map at 5 Hz, it could be just 1 Hz, so you can map 5x more. You also set Rtabmap/TimeThr to 100 ms, which is very small.

The error you are seeing is maybe icp_odometry failed the registration, so odometry is lost. You would have to reset odometry. Doing so it creates a new session. If a loop closure is detected with a previous session, the maps will be merged automatically (like in tango multi-session).

To debug icp_odometry (the drift we see is caused by odometry), it cannot be done just looking at the database, a rosbag of the raw scans is required!

Regards,
Mathieu

[sDeboffle]

hi mathieu
thanks for your answer. That's work fine.
I will try to save you a rosbag for icp_odometry debug and refine a little results.
We plan to minimize our system in order to have more mobility with it.
our laptop is a acer predator G3-572 with I7-7700, 8Go Ram, and gtx 1060 and that work well actually
What is the most important things for mapping? CPU and GPU capacity?

I search and found some solution but I wanted to ask you what do you think is the best solution between:
JETSON TX2
HP Workstation Z2 Mini G4 Performance
Lenovo ThinkCentre M900
if you know other solution that's work with my setup, I'm listening

cheers
stephane

[matlabbe]

Hi,

CPU is the most important thing. There are no optimizations done on GPU for scan matching. GPU is mostly used only for visualization/rendering the map in real-time. A bigger GPU will make the visualization smoother and also details can be increased (more points to show). Same for RAM, larger maps can be visualized. If visualization is done remotely, a GPU is not needed. A SSD is always nice too, as the database is recorded on hard-drive by default. If you have a lot of RAM, you may even keep the whole database in memory instead of hard drive (parameter DbSqlite3/InMemory=true). The disadvantage is that if the computer crashes, there is no way to recover the data (when database is on disk, it is possible to recover the database after a system crash, see "rtabmap-recovery" tool).

For the configuration, it is difficult to say. Sure faster is better in general (to avoid having 100% CPU usage on one core). However, the quality may not be higher depending on the speed of the robot/scanner. For example, icp done in 20 ms at 5 m/s would give same quality than icp done in 100 ms at 1 m/s. Well, depending what is the frame rate of the lidar, you may get better results when moving slower anyway to avoid displacement errors inside a single scan during its rotation. If you can increase the rotation rate of the lidar, I recommend to do so.

cheers,
Mathieu

[sDeboffle]

Hi mathieu.
So...
We reinstall all on a new machine.
that's not work as well as on the old laptop and we don't really understand why.


after 30 image ID, rtabmap start showing this message on new PC:
[ INFO] [1560504701.068338570]: Odom: ratio=0.746964, std dev=0.037634m|0.037634rad, update time=0.099078s

[ WARN] [1560791253.260951213]: Could not get transform from Lidar to Lidar accordingly to odom after 0.100000 seconds (for stamps=1560791252.249110 -> 1560791252.942867)! Error="Lookup would require extrapolation into the future.  Requested time 1560791252.942867056 but the latest data is at time 1560791233.089997952, when looking up transform from frame [odom] to frame [Lidar]. canTransform returned after 0.100979 timeout was 0.1.".

for the same image id, old laptop show:
[ INFO] [1560504701.068338570]: Odom: ratio=0.546964, std dev=0.037634m|0.037634rad, update time=0.040078s


the only difference we see is the average rate of Lidar/points topic (2.5hz on old laptop, 10 on the new PC)but don't know why (probably ethernet ports limit). So I assume RTABmap receive more data on new PC and don't manage them well in 0.1s.

We try all day to increase this 0.1s timout without success.
We change Rtabmap/TimeThr to 0.2.
we tried wait_for_transform to the same value.
we tried to set tf_tolerance too.

nothing works to increase this value.

How I can do?
db: https://we.tl/t-RBocowk3QC

cheers
stephane

[matlabbe]

Hi,

You should not have to set Rtabmap/TimeThr in many cases, unless you want to play with memory management. I would keep it at 0 for debugging the setup first.

What looks like the TF tree in RVIZ? Your image/lidar are tilted a lot!


For the warning, check the update time, 100 ms on new computer and 40 ms on the older one. Are they using the same parameters? Are the point clouds larger on new pc? The frame rate would not affect the odometry time.

If you have a rosbag + launch file used to use the bag, I may take some time to try it.

cheers,
Mathieu