Performance limitations with rtabmap-kitti_dataset tool?

[dave]

Dear Mathieu,

I am trying to use rtabmap on my own dataset in an environment with very few features, and I want to compare different feature detectors and descriptors.

I took some rosbags in ROS 2 and I am able to run rtabmap on the stereo images from the rosbag. However, I noticed that the performance on my laptop is limited, so I reduced the rosbag playback speed to 0.2 to eliminate the performance issues (which consist of losing the odometry).

I also made a script that can extract all the images from the rosbag (I need them later for another project), and I found out about the rtabmap-kitti_dataset tool (I have my data in the same format as the KITTI dataset), which allows me to run rtabmap on these extracted images without using ROS 2 at all. I expected this would also eliminate the performance issues as it should be able to go through the image pairs one-by-one and take all the time it needs to process. However, it also loses the odometry at the same position as with the rosbag at full playback speed.

When it loses the odometry, I get the following message:

```
[ WARN] (2024-12-02 16:21:27.226) Features2d.cpp:898::generateKeypoints3D() A large number (511/1000) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (128.000000) or that exposure between left and right images is too different.
```

1) The images are calibrated and rectified.

2) The background might not always be visible as the robot I use has the camera attached at a downward angle of 30 degrees (since the features to look at are on the ground, not the landscape), and therefore, there might not be a background in the FoV of the camera. Could this be an issue?

3) I tried increasing the maximum disparity value to 256 and even 512, but it still has the same issue.

4) The exposure of the stereo images is the same (to my eyes, at least), so I cannot believe this would be a limiting factor.

Which leads me to my initial question: Are there performance limitations with the rtabmap-kitti_dataset tool? Is there anything that tries to keep up with real-time performance when running this tool compared to running rtabmap on the rosbag data?

I found the parameter Rtabmap/DetectionRate = "1". Is this doing what I think to be forcing the real-time behaviour? I tried setting it to 0.2, but it did not change anything. Does it need to be an integer? What if I set it to 0?

Am I overlooking anything?

With kind regards,
Dave

[dave]

Just for clarification, I use the following command to start rtabmap on my own data with the extracted images:

```
rtabmap-kitti_dataset --Vis/FeatureType 1 --Kp/DetectorStrategy 1 --StereoSGBM/MinDisparity 256 --Stereo/MaxDisparity 512  --Stereo/DenseStrategy 1 --Rtabmap/DetectionRate 0 /home/dave/Desktop/feature_comparison/extracted_images
```

(I tried different values for StereoSGBM/MinDisparity and Stereo/MaxDisparity.)

The folder structure is as follows:

```
- extracted_images
  - images_0
    - 000000.png
    - 000001.png
    - 000002.png
  - images_1
    - 000000.png
    - 000001.png
    - 000002.png
  - calib.txt
  - times.txt
```
Do I maybe forget any other flags that I need to set?

[matlabbe]

Can you share some images for left and right cameras? rtabmap-kitti_dataset is fully offline processing, every frames are processed. The issue you have is not computation issue, it is the lack of good visual correspondences between the images. Lack of texture or bad rectification may cause that issue. In case of lack of texture, there is nothing we can do, other than resetting odometry to continue with next frames. To do so, you can add "--Odom/ResetCountdown 1".

cheers,
Mathieu

[dave]

Dear Mathieu,

Thank you very much for the reply.

I know, that the images are very challenging and that there are only few good features that can be tracked among images. I attached a stereo image pair to show, what kind of images we are dealing with. The point of my research is exactly the fact, that it is difficult.




I am wondering, why the ROS 2 version does not immediately loose the odometry. I suspect, that it does multiple attempts to reset the odometry before it gives up and declares that it lost the odometry, am I right?

When I use the "--Odom/ResetCountdown 1", is there any trace in the output data, that the odometry was lost? It would be good to know, how many times it had to reset its position along a certain trajectory.

Please let me know, if I can provide any further details.

With kind regards,
Dave

P.S.
I'd like not to put the entire dataset online as it has been submitted for review in a journal, and we are waiting for a reply from the reviewers. I could share it with you privately if you like. I could send you a link vie email if you agree with that. Please let me know.