rtabmap_ros nodes and mapping stop, but visual odometry continues

[Jacobmo]

I am running rtabmap_ros in a gazebo simulation environment and every time I run it, it works for a little while, then randomly it stops building the map and generating nodes, but the visual odometry continues, I don't see any errors and the odometry quality seems to stay good the whole time. It wasn't doing this on my other computer, but it was too laggy to run the simultaion. Have you seen that before? and do you know how to make it work?

Here is the printout of the terminal while it is running:

[ INFO] [1529939978.089749399, 824.383000000]: Odom: quality=297, std dev=0.025908m|0.057815rad, update time=0.060999s
[ INFO] [1529939978.253962122, 824.543000000]: Odom: quality=189, std dev=0.047151m|0.073336rad, update time=0.060697s
[ INFO] [1529939978.424092037, 824.710000000]: Odom: quality=550, std dev=0.026971m|0.056322rad, update time=0.063738s
[ INFO] [1529939978.592198204, 824.876000000]: Odom: quality=431, std dev=0.050391m|0.069372rad, update time=0.062679s
[ INFO] [1529939978.659220890, 824.935000000]: rtabmap (40): Rate=1.00s, Limit=0.000s, RTAB-Map=0.0603s, Maps update=0.0011s pub=0.0002s (local map=40, WM=40)
[ INFO] [1529939978.783758750, 825.050000000]: Odom: quality=337, std dev=0.060708m|0.059126rad, update time=0.062468s
[ INFO] [1529939978.959103890, 825.226000000]: Odom: quality=206, std dev=0.025817m|0.063551rad, update time=0.057312s
[ INFO] [1529939979.129505080, 825.394000000]: Odom: quality=566, std dev=0.043899m|0.060841rad, update time=0.061644s
[ INFO] [1529939979.299010714, 825.559000000]: Odom: quality=481, std dev=0.041142m|0.056533rad, update time=0.067617s
[ INFO] [1529939979.456817888, 825.711000000]: Odom: quality=347, std dev=0.029134m|0.058856rad, update time=0.059643s
[ INFO] [1529939979.628609041, 825.882000000]: Odom: quality=266, std dev=0.069334m|0.064364rad, update time=0.062497s
[ INFO] [1529939979.712465231, 825.962000000]: rtabmap (41): Rate=1.00s, Limit=0.000s, RTAB-Map=0.0799s, Maps update=0.0015s pub=0.0003s (local map=41, WM=41)
[ INFO] [1529939979.827479290, 826.075000000]: Odom: quality=178, std dev=0.055426m|0.073868rad, update time=0.046216s
[ INFO] [1529939980.036737401, 826.285000000]: Odom: quality=541, std dev=0.018232m|0.047457rad, update time=0.062829s
[ INFO] [1529939980.236173028, 826.482000000]: Odom: quality=424, std dev=0.024773m|0.064610rad, update time=0.062697s
[ INFO] [1529939980.412927491, 826.655000000]: Odom: quality=291, std dev=0.021599m|0.060275rad, update time=0.072550s
[ INFO] [1529939980.558711465, 826.797000000]: Odom: quality=217, std dev=0.023372m|0.066784rad, update time=0.051396s
[ INFO] [1529939980.752262900, 826.989000000]: Odom: quality=571, std dev=0.044835m|0.050939rad, update time=0.073907s
[ INFO] [1529939980.843790452, 827.073000000]: rtabmap (42): Rate=1.00s, Limit=0.000s, RTAB-Map=0.0718s, Maps update=0.0027s pub=0.0002s (local map=42, WM=42)
[ INFO] [1529939980.946114682, 827.160000000]: Odom: quality=472, std dev=0.013570m|0.057250rad, update time=0.077210s
[ INFO] [1529939981.113646661, 827.330000000]: Odom: quality=369, std dev=0.039251m|0.058357rad, update time=0.059422s
[ INFO] [1529939981.280316454, 827.494000000]: Odom: quality=259, std dev=0.025684m|0.053875rad, update time=0.054166s
[ INFO] [1529939981.483952090, 827.695000000]: Odom: quality=544, std dev=0.054043m|0.060650rad, update time=0.079230s
[ INFO] [1529939981.642723532, 827.848000000]: Odom: quality=461, std dev=0.020721m|0.054194rad, update time=0.057846s
[ INFO] [1529939981.721492538, 827.925000000]: rtabmap (43): Rate=1.00s, Limit=0.000s, RTAB-Map=0.0714s, Maps update=0.0029s pub=0.0002s (local map=43, WM=43)
[ INFO] [1529939981.826151381, 828.027000000]: Odom: quality=338, std dev=0.021702m|0.066229rad, update time=0.063914s
[ INFO] [1529939981.986880687, 828.188000000]: Odom: quality=229, std dev=0.047772m|0.071245rad, update time=0.050682s
[ INFO] [1529939982.175219592, 828.374000000]: Odom: quality=570, std dev=0.081802m|0.078155rad, update time=0.066523s
[ INFO] [1529939982.353096029, 828.549000000]: Odom: quality=468, std dev=0.030128m|0.066464rad, update time=0.069318s
[ INFO] [1529939982.520100285, 828.714000000]: Odom: quality=349, std dev=0.062641m|0.067845rad, update time=0.052500s
[ INFO] [1529939982.695432757, 828.888000000]: Odom: quality=220, std dev=0.022045m|0.057309rad, update time=0.056160s
[ INFO] [1529939982.880592620, 829.070000000]: Odom: quality=563, std dev=0.038152m|0.058845rad, update time=0.064992s
[ INFO] [1529939982.963652743, 829.142000000]: rtabmap (44): Rate=1.00s, Limit=0.000s, RTAB-Map=0.0656s, Maps update=0.0028s pub=0.0003s (local map=44, WM=44)
[ INFO] [1529939983.073320953, 829.244000000]: Odom: quality=505, std dev=0.046060m|0.070382rad, update time=0.070166s
[ INFO] [1529939983.229243701, 829.402000000]: Odom: quality=349, std dev=0.034000m|0.065184rad, update time=0.053953s
[ INFO] [1529939983.402507870, 829.574000000]: Odom: quality=251, std dev=0.028924m|0.072992rad, update time=0.059661s
[ INFO] [1529939983.577150086, 829.747000000]: Odom: quality=549, std dev=0.023917m|0.064163rad, update time=0.067690s
[ INFO] [1529939983.742236648, 829.908000000]: Odom: quality=457, std dev=0.018199m|0.058967rad, update time=0.063153s
[ INFO] [1529939983.905313937, 830.069000000]: Odom: quality=370, std dev=0.033258m|0.061741rad, update time=0.060209s
[ INFO] [1529939984.129968377, 830.288000000]: Odom: quality=248, std dev=0.043463m|0.077642rad, update time=0.071243s
[ INFO] [1529939984.316179421, 830.473000000]: Odom: quality=543, std dev=0.043719m|0.057800rad, update time=0.068542s
[ INFO] [1529939984.474720157, 830.630000000]: Odom: quality=457, std dev=0.028817m|0.063189rad, update time=0.059638s
[ INFO] [1529939984.668474293, 830.821000000]: Odom: quality=313, std dev=0.013510m|0.051188rad, update time=0.074895s
[ INFO] [1529939984.808135572, 830.959000000]: Odom: quality=179, std dev=0.026509m|0.064298rad, update time=0.051650s
[ INFO] [1529939984.979804045, 831.129000000]: Odom: quality=578, std dev=0.016109m|0.062214rad, update time=0.063778s
[ INFO] [1529939985.146755954, 831.296000000]: Odom: quality=493, std dev=0.022732m|0.071816rad, update time=0.059944s
[ INFO] [1529939985.331620863, 831.479000000]: Odom: quality=362, std dev=0.012898m|0.054731rad, update time=0.051903s
[ INFO] [1529939985.544052564, 831.689000000]: Odom: quality=209, std dev=0.024374m|0.064133rad, update time=0.071409s
[ INFO] [1529939985.724055396, 831.866000000]: Odom: quality=552, std dev=0.023940m|0.067598rad, update time=0.060544s
[ INFO] [1529939985.907618716, 832.045000000]: Odom: quality=392, std dev=0.016572m|0.057617rad, update time=0.056194s
[ INFO] [1529939986.102247588, 832.236000000]: Odom: quality=247, std dev=0.015963m|0.056441rad, update time=0.077555s
[ INFO] [1529939986.278591890, 832.408000000]: Odom: quality=547, std dev=0.009361m|0.059150rad, update time=0.079314s
[ INFO] [1529939986.425049710, 832.552000000]: Odom: quality=387, std dev=0.008655m|0.049069rad, update time=0.057852s
[ INFO] [1529939986.583538571, 832.707000000]: Odom: quality=249, std dev=0.035012m|0.072163rad, update time=0.058107s
[ INFO] [1529939986.793065444, 832.913000000]: Odom: quality=508, std dev=0.022844m|0.059949rad, update time=0.078925s
[ INFO] [1529939986.936879628, 833.055000000]: Odom: quality=285, std dev=0.059347m|0.053961rad, update time=0.052382s
[ INFO] [1529939987.110277098, 833.224000000]: Odom: quality=242, std dev=0.027616m|0.055227rad, update time=0.052501s
[ INFO] [1529939987.282295099, 833.393000000]: Odom: quality=466, std dev=0.017907m|0.049201rad, update time=0.060854s
[ INFO] [1529939987.452033567, 833.559000000]: Odom: quality=354, std dev=0.051183m|0.051957rad, update time=0.060601s
[ INFO] [1529939987.640005670, 833.745000000]: Odom: quality=263, std dev=0.024157m|0.062714rad, update time=0.072639s
[ INFO] [1529939987.828524468, 833.929000000]: Odom: quality=493, std dev=0.027306m|0.053208rad, update time=0.085726s
[ INFO] [1529939987.994795053, 834.094000000]: Odom: quality=409, std dev=0.036003m|0.057289rad, update time=0.083847s
[ INFO] [1529939988.147159405, 834.245000000]: Odom: quality=297, std dev=0.029683m|0.063549rad, update time=0.060412s
[ INFO] [1529939988.322324331, 834.416000000]: Odom: quality=212, std dev=0.043852m|0.076757rad, update time=0.070383s
[ INFO] [1529939988.520859795, 834.615000000]: Odom: quality=531, std dev=0.016761m|0.069561rad, update time=0.085876s
[ INFO] [1529939988.678085338, 834.769000000]: Odom: quality=465, std dev=0.019660m|0.057816rad, update time=0.072243s
[ INFO] [1529939988.850631472, 834.939000000]: Odom: quality=371, std dev=0.022885m|0.055956rad, update time=0.078577s
[ INFO] [1529939989.041581580, 835.127000000]: Odom: quality=266, std dev=0.024896m|0.055714rad, update time=0.065424s

so it is generating the nodes and then all of a sudden the  

rtabmap (41): Rate=1.00s, Limit=0.000s, RTAB-Map=0.0799s, Maps update=0.0015s pub=0.0003s (local map=41, WM=41)

 part stops and only odometry works.

It happens at different times and places every time, and it never starts working again once it stops.

I did some additional testing and it seems to do that when it should have found a positive loop closure. It doesn't say that, but it fails most often right as the robot is completing the first loop.

I have it installed with g20 and GTSAM.

Any advice would help.

Thank You,
Jacob

[matlabbe]

Hi Jacob,

rtabmap node should always output a status at 1 Hz. If not, either the input topics cannot be synchronized or the node is crashed. As visual odometry is working, I suspect more a crash. Can you see a "rtabmap process has died" error on terminal at some point?

Can you try setting parameter Optimizer/Strategy of rtabmap node to 0 (for TORO optimization)? If not crashing anymore on loop closure, this could be a g2o/gtsam optimization problem. Which version of PCL is rtabmap built against? PCL 1.8 builds with -march=native flag, which is causing Eigen problem when building against a version of GTSAM or g2o built without this flag.

cheers,
Mathieu

[Jacobmo]

Just getting back to this, I changed gears for a little bit since it wasn't working.

yeah it seems like a crash, but the "rtabmap process has died" never prints.

I switched the optimizer strategy to 0 for toro and the problem stopped, the Issue I am running into with toro is that it often detects false loop closures which is why I installed it with GTSAM and g2o.

I checked the version of PCL and it says 1.7. Would updating it to 1.8 help?

[matlabbe]

With PCL 1.7, the flag "-march=native" is not set by default, so nothing special would have to be done when building g2o or gtsam. Without any errors, it is difficult to debug the problem. Did you try launching rtabmap with gdb (see this)?