How to/ On What basis to compare different slam like you did in your VSLAM paper?

[BalajiRavi]

Dear Matheiu,
Thanks for your continous support.

(For outdoor agriculture field)

1.  I am doing a project paper where i am comparing different SLAMS using D435i, I read your paper where you have compared different SLAM with RTABSLAM using KITTI and TUM datasets. May I know on what basis to compare them? Like how to record and compare which values?

2. Also I am planning to compare slam in D435i vs Livox Mid360. How can i compare both? Using point cloud or could you give your insights on this.

Setup : I already have setup with D435i working with RTABMAP and its fine. Also have livoxmid360 with its drivers installed but not slam.

It would be really helpful for my project paper. Thanks in advance

Regards,
Balaji

[matlabbe]

In that paper, we compared the trajectory estimated against a "ground truth" trajectory, using either KITTI proposed metrics or TUM RGB-D dataset proposed metrics. All those cases require an external system to give accurate global localization (such as VICON, optitrack for indoor, or GPS-RTK / IMU fusion for outdoor). Using trajectory to estimate SLAM accuracy is the most common approach.

Another way is to compare to a floor plan, or overlaying against Google Earth satellite images to assess the quality (though it is more a qualitative comparison than quantitative like the previous approach).

If you cannot estimate the trajectory accurately, another way is to compare the resulting point clouds of a SLAM approach to a "ground truth" point cloud, created for example from a 3D total station surveying. CloudCompare app can then help to estimate how similar the point clouds are.

In your case, you are gonna compare a stereo point cloud against a LiDAR point cloud. Indoor I would say LiDAR wins, but for outdoor with a lot of texture, they may be not too far, though stereo point clouds are always more noisier. When using two modalities like that, you wold have to assess if the difference is caused by trajectory difference (e.g., they are not using the same odometry approach) or that the sensor's point cloud is different (e.g., in case they used exactly the same trajectory).

cheers,
Mathieu

[BalajiRavi]

Thank you so much for a detailed answer. If I’m planning to use my own rosbag dataset of an agriculture field, i need ground truth data, can you advise how to create a ground truth data

[BalajiRavi]

Also I also read in a research paper, ground truth can be measured using LIDAR too. in my case is it possible with a livox mid-360 lidar?

[matlabbe]

ground truth can be measured using LIDAR too

SLAM with lidar doesn't give a real ground truth, but it some cases it can pretty close. You will still need a way to assess that the trajectory and point cloud generated by the lidar SLAM approach is close to real world. One way to do that would be to use a surveying service to survey the area (with a total station) you are testing, then see if the lidar point cloud is aligned with control points. If you are referring to PR2 MIT dataset in one of my paper, they used a 2D lidar slam to get a first look at the trajectory, then manually corrected errors if there were to align perfectly with the building plan.

Like I said in my first post, you may use a GPS RTK (or differential GPS) product to do so. To do it yourself (DIY):
https://www.ardusimple.com/how-to-use-ardusimple-rtk-receivers-and-get-gps-data-in-ros/ 
https://www.ardusimple.com/product/simplertk2b-starter-kit-lr-ip65/ (latitude-longitude-altitude)
https://www.ardusimple.com/product/simplertk2b-heading-basic-starter-kit-ip67/ (latitude-longitude-altitude-heading)

If money is not an issue (>10k$), you may get a product like this https://store.clearpathrobotics.com/collections/gps/products/duro-rtk-package (ros driver https://github.com/szenergy/duro_gps_driver)