rtabmap node crashes when I try to approx_sync icp_odometry and rgb camera

[GilMe]

For my setup I have a solid state lidar (that generates a 3d pointcloud) and a rgb camera.
In my launch file I generate odometry using an icp_odometry node that uses data coming from the lidar. I then feed this odometry to a rtabmap node along with subscribing to my rgb camera and lidar. If the approx_sync varible is set to false i get the following error:

[ WARN] [1666880020.278087704, 1666096551.027407952]: /rtabmap/rtabmap: Did not receive data since 5 seconds! Make sure the input topics are published ("$ rostopic hz my_topic") and the timestamps in their header are set. If topics are coming from different computers, make sure the clocks of the computers are synchronized ("ntpdate"). Parameter "approx_sync" is false, which means that input topics should have all the exact timestamp for the callback to be called.
/rtabmap/rtabmap subscribed to (exact sync):
   /rtabmap/scanmatch_odom \
   /camera/image_raw_relay \
   /camera/camera_info \
   /lidar/scan_3D \
   /rtabmap/odom_info


If I set the approx_sync variable to true the rtabmap node crashes even when I only subscribe to the rgb topic. It gives me the following error:

[rtabmap/rtabmap-5] process has died [pid 14307, exit code -8, cmd /home/cs3/new_nano_ws/devel/lib/rtabmap_ros/rtabmap --delete_db_on_start /data_throttled_image/compressed:=/camera/image_raw/compressed rgb/image:=/camera/image_raw_relay rgb/camera_info:=/camera/camera_info scan_cloud:=/lidar/scan_3D odom:=scanmatch_odom odom_info:=odom_info __name:=rtabmap __log:=/home/cs3/.ros/log/b54bfff4-5601-11ed-a20c-d41b81c98bb3/rtabmap-rtabmap-5.log].
log file: /home/cs3/.ros/log/b54bfff4-5601-11ed-a20c-d41b81c98bb3/rtabmap-rtabmap-5*.log

I have checked and it also happens when I don't subscribe to lidar (when subscribe_scan_cloud is set to false).
What am I doing wrong?

[matlabbe]

It is likely that your camera and lidar don't publish with exact same stamp, so approx_sync should be true.

For the crash, try section A of this https://github.com/introlab/rtabmap_ros/issues/28, this will give more info where/why it crashes. The node can also be launched with "--udebug" argument to show debug logs if it is not crashing on start (which would mean a loading library problem).

[GilMe]

First of all thanks your quick reply.

I ran gdb as for your suggestion. Apparently the issue doesn't have anything to do with sync. The crash happens in the convertRGBDMsgs function in the MsgConversion.cpp
The program crashes in this part of the code (lines 1859-1864):

if(isDepth)
        {
                UASSERT_MSG(
                                imageWidth/depthWidth == imageHeight/depthHeight,
                                uFormat("rgb=%dx%d depth=%dx%d", imageWidth, imageHeight, depthWidth, depthHeight).c_str());
        }

Since the message doesn't have a depth image than depthWidth = 0 and I get an arithmetic exception.

I tried to investigate why isDepth is true and got to lines 1835-1838 where is defined:

bool isDepth = depthMsgs.empty() || (depthMsgs[0].get() != 0 && (
                        depthMsgs[0]->encoding.compare(sensor_msgs::image_encodings::TYPE_16UC1) == 0 ||
                        depthMsgs[0]->encoding.compare(sensor_msgs::image_encodings::TYPE_32FC1) == 0 ||
                        depthMsgs[0]->encoding.compare(sensor_msgs::image_encodings::MONO16) == 0));

I thought that there might be a bug and should be:

bool isDepth = !depthMsgs.empty() && (depthMsgs[0].get() != 0 && (
                        depthMsgs[0]->encoding.compare(sensor_msgs::image_encodings::TYPE_16UC1) == 0 ||
                        depthMsgs[0]->encoding.compare(sensor_msgs::image_encodings::TYPE_32FC1) == 0 ||
                        depthMsgs[0]->encoding.compare(sensor_msgs::image_encodings::MONO16) == 0));

but when I fixed and ran the changed code, I got the following warning " We cannot estimated the baseline of the rectified images with tf! (/camera_link->/camera_link = xyz=0.000000,0.000000,0.000000 rpy=0.000000,-0.000000,0.000000)", and it doesn't really work.

I then saw that if "isDepth" is false than it tries to calculate stereo.

I wanted to ask. I am trying to run rtabmap with lidar and a normal camera. I used the "subscribe_rgb". Why is it trying to get depth images? Why is it sent to convertRGBDMsgs function?

[matlabbe]

Should be fixed in https://github.com/introlab/rtabmap_ros/commit/ccd7b6272117f81148030259c886b11b99fdc527, thx for investigating!