Stereo odometry warns on euroc dataset

[JRL]

Hi there!
I am using stereo odometry of rtabmap_ros as a part of my little project, however, when I try to test the odometry on euroc dataset, it always warns me that

<code>
[stereo_odometry-1] [ WARN] (2024-06-28 21:46:09.505) Features2d.cpp:837::generateKeypoints3D() A large number (790/937) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (256.000000) or that exposure between left and right images is too different.
[stereo_odometry-1] [ WARN] (2024-06-28 21:46:09.544) OdometryF2M.cpp:557::computeTransform() Registration failed: "Not enough inliers 0/8 (matches=72) between -1 and 11894"
</code>

I am new to SLAM, but I tried to calibrated the euroc images with intrinsics and extrinsics in Pyhon:

<code>
dist_coeffs_left = np.array([-0.28340811, 0.07395907, 0.00019359, 1.76187114e-05])
camera_matrix_left = np.array([[458.654, 0, 367.215],
                          [0, 457.296, 248.375],
                          [0, 0, 1]])
dist_coeffs_right = np.array([-0.28368365,  0.07451284, -0.00010473, -3.55590700e-05])
camera_matrix_right = np.array([[457.587, 0, 379.999],
                          [0, 456.134, 255.238],
                          [0, 0, 1]])
left_folder="/home/dumas/mav0/cam0/data/"
right_folder="/home/dumas/mav0/cam1/data/"
rect_left_folder="/home/dumas/mav0_rect/cam0/data/"
rect_right_folder="/home/dumas/mav0_rect/cam1/data/"
height = 480
weight = 752
new_camera_matrix_left, roi_left = cv2.getOptimalNewCameraMatrix(camera_matrix_left, dist_coeffs_left, (weight,height), 0, (weight,height))
new_camera_matrix_right, roi_right = cv2.getOptimalNewCameraMatrix(camera_matrix_right, dist_coeffs_right, (weight,height), 0, (weight,height))
all_file=os.listdir(left_folder)

for file in all_file:
    if file.endswith(".png"):
        distorted_image_left = cv2.imread(left_folder+file,cv2.IMREAD_GRAYSCALE)
        distorted_image_right = cv2.imread(right_folder+file,cv2.IMREAD_GRAYSCALE)
        undistorted_image_left = cv2.undistort(distorted_image_left, camera_matrix_left, dist_coeffs_left, None, new_camera_matrix_left)
        undistorted_image_right = cv2.undistort(distorted_image_right, camera_matrix_right, dist_coeffs_right, None, new_camera_matrix_right)
        # exposure compensation
        avg_brightness_left = np.mean(undistorted_image_left)
        avg_brightness_right = np.mean(undistorted_image_right)
        diff_brightness = avg_brightness_right - avg_brightness_left
        adjusted_image_left = cv2.convertScaleAbs(undistorted_image_left, alpha=1, beta=diff_brightness)  
        # save images
        cv2.imwrite(rect_left_folder+file, adjusted_image_left)
        cv2.imwrite(rect_right_folder+file, undistorted_image_right)
</code>

And I got calibrated images like this:
left_image_rectified:

right_image_rectified:

_________________________________________________________________________________
The raw images are here:
left_image_raw:

right_image_raw:


It's hard for me to handle this problem, could you give me some advices? looking forward to hearing from you, thanks so much!

[matlabbe]

Hi,

The rectified images are not stereo rectified (there is a vertical disparity):


You should use stereoRectify() function to get the optimal new camera matrices for each camera.



cheers,
Mathieu

[JRL]

Hi Mathieu! Thanks to your reply, it really helps a lot.
At this time, I used  stereoRectify()-->initUndistortRectifyMap()-->remap() function in cv2 to rectified euroc images, and the output of stereo odometry seems to be much more normal.
However, I still got warnings sometime like this:
[stereo_odometry-1] [INFO] [1719852254.959861299] [r0.stereo_odometry]: Odom: quality=321, std dev=0.007832m|0.016214rad, update time=0.000000s
[stereo_odometry-1] [INFO] [1719852255.089014543] [r0.stereo_odometry]: Odom: quality=323, std dev=0.006500m|0.014651rad, update time=0.000000s
[stereo_odometry-1] [INFO] [1719852255.244586115] [r0.stereo_odometry]: Odom: quality=295, std dev=0.008659m|0.016542rad, update time=0.000000s
[stereo_odometry-1] [INFO] [1719852255.359294387] [r0.stereo_odometry]: Odom: quality=295, std dev=0.007256m|0.015492rad, update time=0.000000s
[stereo_odometry-1] [INFO] [1719852255.519352875] [r0.stereo_odometry]: Odom: quality=312, std dev=0.008153m|0.015492rad, update time=0.000000s
[stereo_odometry-1] [ WARN] (2024-07-02 00:39:28.209) Features2d.cpp:837::generateKeypoints3D() A large number (530/905) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (256.000000) or that exposure between left and right images is too different.
[stereo_odometry-1] [ WARN] (2024-07-02 00:39:28.346) Features2d.cpp:837::generateKeypoints3D() A large number (541/912) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (256.000000) or that exposure between left and right images is too different.
[stereo_odometry-1] [ WARN] (2024-07-02 00:39:28.498) Features2d.cpp:837::generateKeypoints3D() A large number (518/908) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (256.000000) or that exposure between left and right images is too different.
[stereo_odometry-1] [ WARN] (2024-07-02 00:44:12.345) Features2d.cpp:837::generateKeypoints3D() A large number (469/923) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (256.000000) or that exposure between left and right images is too different.
[stereo_odometry-1] [ WARN] (2024-07-02 00:44:12.483) Features2d.cpp:837::generateKeypoints3D() A large number (482/915) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (256.000000) or that exposure between left and right images is too different.
[stereo_odometry-1] [ WARN] (2024-07-02 00:44:12.783) Features2d.cpp:837::generateKeypoints3D() A large number (459/917) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (256.000000) or that exposure between left and right images is too different.
[stereo_odometry-1] [ WARN] (2024-07-02 00:44:13.783) Features2d.cpp:837::generateKeypoints3D() A large number (474/926) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (256.000000) or that exposure between left and right images is too different.
[stereo_odometry-1] [ WARN] (2024-07-02 00:44:15.212) Features2d.cpp:837::generateKeypoints3D() A large number (471/928) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (256.000000) or that exposure between left and right images is too different.
[stereo_odometry-1] [ WARN] (2024-07-02 00:44:15.327) Features2d.cpp:837::generateKeypoints3D() A large number (472/919) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (256.000000) or that exposure between left and right images is too different.
[stereo_odometry-1] [ WARN] (2024-07-02 00:44:15.489) Features2d.cpp:837::generateKeypoints3D() A large number (518/905) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (256.000000) or that exposure between left and right images is too different.

Seems that the odom can't create 3D keypoints because of unrectified images, right?
When I reviewed the rectified images, I got left image and right image like this:
rectified left image:

rectified right image:

I wonder if these two images are rectified? These two images seem to have different perspective.
If not, could you give me some advices to make them rectified?
Thanks in advance.
btw, the stereo odom should publish "odom"/"odom info"/"odom_last_frame" topics according to rtabmap document, but I can't see these topics using "ros2 topic list" command. How can I fix this problem?

[matlabbe]

Th rectified images look better:


These warnings are normal in this part of the sequence. The perspective difference is too large (close objects to camera) to find stereo correspondences at the bottom half of the image. As the warning says, that "could" be fixed by increasing maximum disparity (which is 256). I say "could" because the perspective may be too different to actually see exactly the same visual feature in both images. In other words, you could ignore these warnings. After take-off, these warnings should disappear.

Do a "ros2 topic list | grep odom" to see the actual topic, they may be under a namespace.

cheers,
Mathieu