RTABMAP closes with error when mapping for long durations

[alexr]

Hi Mathieu,

I am having this trouble when using rtabmap with the robot and mapping for long durations. I was trying to make a loop for long corridors (approx 40m x 4 ). The loop closes nicely but when I ctrl-c the terminal where the rtabmap is running, it exits with a error messages. I get the following errors:

[ INFO] [1422700213.490803637]: rtabmap: Update rate=0.500000s, Limit=0.000000s, Processing time = 0.420058s (1543 local nodes)
[mobile_base_nodelet_manager-25] killing on exit
 [turtlebot_teleop_joystick-27] killing on exit
[rviz-24] killing on exit
[joystick-28] killing on exit
[ WARN] (2015-01-31 19:30:14.017) Rtabmap.cpp:1335::process() Cannot compute a loop closure transform between 2479 and 2489: Not enough inliers 0/5 between 2479 and 2489
[ INFO] [1422700214.021549977]: rtabmap: Update rate=0.500000s, Limit=0.000000s, Processing time = 0.351387s (1543 local nodes)
[teleop_velocity_smoother-26] killing on exit
[rtabmap/grid_map_assembler-23] killing on exit
[hokuyo-21] killing on exit
[rtabmap/rtabmap-22] killing on exit
[ WARN] (2015-01-31 19:30:18.770) Rtabmap.cpp:1335::process() Cannot compute a loop closure transform between 2479 and 2489: Not enough inliers 0/5 between 2479 and 2489
[INFO] [WallTime: 1422700228.082291] Mapper Finished
[soundplay-31] escalating to SIGTERM
[rviz-24] escalating to SIGTERM
[turtlebot_teleop_joystick-27] escalating to SIGTERM[mapper-30] escalating to SIGTERM

 [rtabmap/rtabmap-22] escalating to SIGTERM
 [soundplay-31] escalating to SIGKILL
[teleop_velocity_smoother-26] escalating to SIGTERM
[hokuyo-21] escalating to SIGTERM
[rtabmap/grid_map_assembler-23] escalating to SIGTERM
[joystick-28] escalating to SIGTERM
[mobile_base_nodelet_manager-25] escalating to SIGTERM
[camera_base_link3-20] killing on exit
[camera_base_link2-19] killing on exit
[camera_base_link1-18] killing on exit
[camera_base_link-17] killing on exit
[camera/points_xyzrgb_hw_registered-16] killing on exit
[camera/depth_registered_rectify_depth-15] killing on exit
[camera/depth_points-14] killing on exit
[camera/depth_metric-13] killing on exit
[camera/depth_metric_rect-12] killing on exit
[camera/depth_rectify_depth-11] killing on exit
[ INFO] [1422700235.800179809]: Stopping depth stream.
[camera/rectify_color-10] killing on exit
[camera/driver-9] killing on exit
[camera/camera_nodelet_manager-8] killing on exit
[relay2-7] killing on exit
[relay1-6] killing on exit
[RosAria-5] killing on exit
[static_transform_publisher3-4] killing on exit
[static_transform_publisher2-3] killing on exit
[static_transform_publisherx-2] killing on exit
[robot_state_publisher-1] killing on exit
[camera/camera_nodelet_manager-8] escalating to SIGTERM
Shutdown errors:
 * process[soundplay-31, pid 2545]: required SIGKILL. May still be running.
shutting down processing monitor...
... shutting down processing monitor complete
done

The rtabmap.db that gets saved in /.ros/rtabmap.db is only 14.3 kb and all the data is lost.
The total number of nodes generated were around 1500. This happened with me on two occasions. What might be the reason and how do I resolve it.?

Thanks

Alex

[matlabbe]

How much RAM memory is used before you kill the processes? As I can see, many processes crashed (many "escalating to SIGTERM"). Maybe your RAM was full.

If the database is almost empty, it is because it is in RAM (saved on hard drive on exit) and/or memory management is disabled (which keeps all locations in Working Memory, [in RAM]). So when rtabmap crashes, nothing is saved.

For large-scale mapping, I recommend to use database on hard drive and set a real-time or memory limit.
To set database on hard drive (not required to set explicitly since it is the default):

<param name="DbSqlite3/InMemory" type="string" value="false"/>

One of these two thresholds:Time Threshold (ms) or  Memory Threshold (maximum nodes in Working Memory)

<param name="Rtabmap/TimeThr" type="string" value="700"/>
<param name="Rtabmap/MemoryThr" type="string" value="400"/>

[alexr]

Dear Mathieu,

Thanks for your reply. I see now the problem. If I am correct rtabmap uses a short term working memory and long term memory for storing the poses and retrieve them from the long term memory when loop closing is required. So by using the suggested parameters I will be forcing them to store only a certain threshold nodes, am i correct?

right now my parameter setting looks like this :

<group ns="rtabmap">

        <node name="rtabmap" pkg="rtabmap" type="rtabmap" output="screen" args="--delete_db_on_start">

            <param name="frame_id" type="string" value="base_link"/>

            <param name="subscribe_depth" type="bool" value="true"/>
            <param name="subscribe_laserScan" type="bool" value="true"/>

            <remap from="odom" to="/odom"/>
            <remap from="scan" to="/scan"/>

            <remap from="rgb/image" to="/camera/rgb/image_rect_color"/>
            <remap from="depth/image" to="/camera/depth_registered/image_raw"/>
            <remap from="rgb/camera_info" to="/camera/depth_registered/camera_info"/>

            <param name="queue_size" type="int" value="10"/>
           <param name="Rtabmap/DatabasePath" type="string" value="$(arg HOME)/.ros/rtabmap.db"/>  <!-- Database used for mapping -->

        <param name="queue_size" type="int" value="10"/>
        
          <!-- RTAB-Map's parameters: do "rosrun rtabmap rtabmap (double-dash)params" to see the list of available parameters. -->
          <param name="Mem/IncrementalMemory" type="string" value="true"/>          <!-- true = Mapping/SLAM mode-->
  
          <param name="Rtabmap/DetectionRate" type="string" value="2"/>     <!-- Don't need to do relocation very often! Though better results if the same rate as when mapping. -->
          <param name="DbSqlite3/CacheSize" type="string" value="10000"/>           <!-- Sqlite cache size (default is 2000) -->
          <param name="RGBD/ScanMatchingSize" type="string" value="0"/>             <!-- Do odometry correction with consecutive laser scans -->
          <param name="RGBD/LocalLoopDetectionSpace" type="string" value="false"/>   <!-- Local loop closure detection (using estimated position) with locations in WM -->
          <param name="RGBD/LocalLoopDetectionTime" type="string" value="false"/>    <!-- Local loop closure detection with locations in STM -->
          <param name="Mem/BadSignaturesIgnored" type="string" value="false"/>      <!-- Don't ignore bad images for 3D node creation (e.g. white walls) -->
          <param name="LccIcp/Type" type="string" value="2"/>                       <!-- Loop closure transformation refining with ICP: 0=No ICP, 1=ICP 3D, 2=ICP 2D -->
          <param name="LccIcp2/Iterations" type="string" value="100"/>
          <param name="LccIcp2/VoxelSize" type="string" value="0"/>
          <param name="LccBow/MinInliers" type="string" value="5"/>                 <!-- 3D visual words minimum inliers to accept loop closure -->
          <param name="LccBow/MaxDepth" type="string" value="4.0"/>                 <!-- 3D visual words maximum depth 0=infinity -->
          <param name="LccBow/InlierDistance" type="string" value="0.05"/>           <!-- 3D visual words correspondence distance -->
          <param name="RGBD/AngularUpdate" type="string" value="0.01"/>             <!-- Update map only if the robot is moving -->
          <param name="RGBD/LinearUpdate" type="string" value="0.01"/>              <!-- Update map only if the robot is moving -->
          <param name="Rtabmap/TimeThr" type="string" value="0"/>
          <param name="Mem/RehearsalSimilarity" type="string" value="0.45"/>
          <param name="Mem/RehearsedNodesKept" type="string" value="false"/>   
          <param name="RGBD/OptimizeFromGraphEnd" type="string" value="false"/>    <!-- Optimize graph from the latest node -->
         <param name="Kp/RoiRatios" type="string" value="0.0 0.0 0.0 0.3"/>    
        </node>

        <node pkg="rtabmap" type="grid_map_assembler" name="grid_map_assembler" output="screen">
                <remap from="grid_map" to="/map"/>
                <param name="scan_voxel_size" type="str" value="0.01" />
        </node>
    </group>

    <group ns="TT">
            <!-- RTAB-Map's parameters: do "rosrun rtabmap rtabmap (double-dash)params" to see the list of available parameters. -->


            <param name="Rtabmap/DetectionRate" type="string" value="3"/>            <!-- Don't need to do relocation very often! Though better results if the same rate as when mapping. -->

            <param name="Mem/STMSize" type="string" value="20"/>                     <!-- 20 location in short-term memory -->
            <param name="Mem/IncrementalMemory" type="string" value="true"/>         <!-- true = mapping mode-->
            
        <param name="Mem/STMSize" type="string" value="20"/>                     <!-- 20 location in short-term memory -->
        <param name="Mem/IncrementalMemory" type="string" value="true"/>         <!-- true = mapping mode-->

        <param name="RGBD/ScanMatchingSize" type="string" value="1"/>            <!-- Do odometry correction with consecutive laser scans -->
        <param name="RGBD/LocalLoopDetectionSpace" type="string" value="false"/> <!-- Local loop closure detection (using estimated position) with locations in WM -->
        <param name="RGBD/LocalLoopDetectionTime" type="string" value="true"/>   <!-- Local loop closure detection with locations in STM -->
        <param name="RGBD/OptimizeFromGraphEnd" type="string" value="false"/>    <!-- Optimize graph from initial node so /map -> /odom transform will be generated -->

        <param name="LccIcp/Type" type="string" value="2"/>                      <!-- 0=No ICP, 1=ICP 3D, 2=ICP 2D -->

        <param name="LccBow/MaxDepth" type="string" value="0.0"/>                <!-- 3D visual words maximum depth 0=infinity -->
        <param name="LccBow/InlierDistance" type="string" value="0.1"/>          <!-- 3D visual words correspondence distance -->
        <param name="LccBow/MinInliers" type="string" value="10"/>

        <param name="Odom/MinInliers" type="string" value="10"/>

    </group>

What is the use of this parameter:

<param name="DbSqlite3/CacheSize" type="string" value="10000"/>  

Can you explain what the values suggested for Rtabmap/TimeThr = 700 and Rtabmap/MemoryThr= 400 are meant for?

Thanks

Alex

[matlabbe]

When the thresholds "Rtabmap/TimeThr" and "Rtabmap/MemoryThr" are not set, nothing is saved to Long-Term Memory (database) until the node is closed. See Section III-F Transfer of the related paper for a description of the "Rtabmap/TimeThr" parameter, referred as T_time in the paper. The "Rtabmap/MemoryThr" parameter is provided for convenience, it has the same effect than T_time, though explicitly fixing the maximum Working Memory size independently of the computing resources.

More practically, I prefer use the Time Threshold (ms). If the time threshold is set to 700 ms and the detection rate is set to 1 Hz, you would have ~70% CPU usage on large environments and stable RAM memory usage. For example:

<param name="Rtabmap/DetectionRate" type="string" value="1"/>
<param name="Rtabmap/TimeThr" type="string" value="700"/>

Here some notes about your launch file:

• "DbSqlite3/CacheSize" doesn't need to be set. See this page for more info.
• "Rtabmap/DetectionRate" is set to 3 Hz, I usually use 1 Hz.
• "Kp/RoiRatios" should not be set unless you want to ignore the some part of the image from features extraction (here you ignore 30% of the bottom image).
• "RGBD/ScanMatchingSize" is now known as "RGBD/PoseScanMatching"
• Parameters in your group "TT" are not used.

You can always see description of the parameters by doing:

$ rosrun rtabmap_ros rtabmap --params

[alexr]

Thank you for your detailed answer. Because of repetitive pattern on the floor, I have filtered the data from the bottom.
 I will try your suggestions and reply back.

Alex

[alexr]

Dear Mathieu,

I tried your suggestions, now I am having a new issue, I changed the detection rate to 1 hz, and added the two parameters for TimeThr and MemoryThr as suggested by you. When running the rtabmap on the robot, when I bring up the rviz, the map keeps deleting the previously recorded data after few minutes, and I can only visualize a part of the map that is currently being built. Even when I download all the clouds from the MapCloud rviz panel, it only shows the present map and do not show all the clouds.

What is the reason for this? I have 8 GB of RAM on an i5 intel system using xtion pro live and a hokuyo urg running on a p3dx pioneer mobile robot.

Thanks

Alex

[matlabbe]

No need to use both thresholds, try without "Rtabmap/MemoryThr", only "Rtabmap/TimeThr".

It is normal that at some point, the oldest locations of the map disappear. RTAB-Map "forgets" some locations to be able to respect any of these two thresholds (you can look at this paper for more detailed information). So only the local map is shown online in RVIZ. To see the global map (all clouds), you must pause rtabmap 

$ rosservice call /rtabmap/pause

and click on "Download graph" under the MapCloud Display in RVIZ.



To resume mapping (old clouds should disappear again since the local map is published again):

$ rosservice call /rtabmap/resume

With "Rtabmap/TimeThr" parameter, the local map size is limited by the time used to process new data in rtabmap. Since you are using an URG, you can reduce the number of features extracted from the images (default is 400) by setting the following parameters.

<param name="Kp/WordsPerImage" type="string" value="200"/>
<param name="SURF/HessianThreshold" type="string" value="1000"/>

This should help to have a bigger local map by reducing processing time used by rtabmap. Note that you can see the processing time used on the console (ROS_INFO) for each rtabmap's iteration.


And here some optimizations to have a better map quality (only use this with a 2D laser):

<param name="RGBD/PoseScanMatching" type="string" value="true"/>           
<param name="RGBD/LocalLoopDetectionSpace" type="string" value="true"/>

[alexr]

Hi Mathieu,

Thanks for your quick reply. I will try this tomorrow and get back to you.

Kind Regards
Alex

[alexr]

Dear Mathieu,

I checked the mapping with new parameters and its improved a lot. Thanks for your suggestions.

Alex