RGB and Depth image as source for SLAM

[cbalthasart]

Hello,

I've integrated rtabmap inside my application using librtabmap_core.so.0.15.3 and librtabmap_utilite.so.0.15.3

I've developed a specific Camera driver to use with rtabmap (like realsense driver or rgbd driver). It is based on the RGBD driver but instead of finding its depth and color images from files inside folders, this images are sent by an external simulated camera (from a unity system with a 3d scene). The system works fine and can create a 3dmap of the scene shown in the unity system. But I've 2 problems :

1. The 3dmap is not exactly the same as the unity scene. I can see that the 3dmap seems distorded (see pictures) against the unity scene (see one of the rgb and depth images sent). I have to provide a "calibration parameterset" to my driver : focal length (fx, fy) and optical center (cx, cy). I think the problem is here. I have no idea how I can obtain them because the camera is a unity camera and unity don't provide this kind of parameters. Rtabmap (standalone) offers a way to calibrate a camera using a chessboard but here the chessboard should be inside the unity scene. Not very easy to do. Is there a possibility to calibrate the camera without chessboard ?

2. The unity scene is located in X=-30 Y=-120 Z=-25 so I've set this coordinates to the localTransform of my Camera driver creation (same has your drivers) : Transform opticalRotation(x,y,z,roll,pitch,yaw); Using this parameters the scene is rendered in the good location (X,Y and Z are near my starting point (X=-30 Y=-120 Z=-25)) but the localization provided by the odometry is not correct. It starts from X=0, Y=0, Z=0 and derives fastly to incoherent values. I've hoped that the localization starts from my starting position and not from 0,0,0 ?

Thanks,

Christophe

Here is the 3dmap rendered by rtabmap.


Here are one of the depth and color image used (from unity)

[matlabbe]

Hi,

1. For the calibration, you can set arbitrary values like fx=fy=525 (like the kinect) with cx=cy=0. The focal distance affects only the scale of the model and trajectory (the horse would appear 10 cm tall or 10 meters tall). You may guess a value to match with Unity scale.

2. Odometry always starts at (0,0,0). In code, we can initialize the odometry at a fixed position (with Odometry::reset()). With the standalone, there is no option to do that, so make sure at least the camera is starting parallel to ground so that the map is not rotated (if the the camera is seeing at least the ground, we can enable Odometry->"Align odometry with ground on initialization." to initialize camera pose with correction orientation and height).

Do you have a resulting database to share? It could be easier to see the problem. In particular, not that rtabmap coordinates are x->forward, y->left, z->up, while Unity uses x->right, y->up, z->forward. So if you try to compare unity poses with rtabmap poses, they won't correspond. We would have to transform the poses into Unity coordinates system to better compare.

cheers,
Mathieu

[cbalthasart]

Hello,

Thanks for your feedback.

I have not kept the data for that test and I'm not able to do it again because I've changed something on the Unity's side.

So I have done another test, still with my famous horse. I've used the parameters fx=334, fy=226 and cx=cy=0. When model is updated with the bgr image size (not known before first image is received), the model recomputes automatically cx to bgr width / 2.0 - 0.5 and cy to bgr height /2.0 - 0.5 (see CameraModel.cpp inside corelib of rtabmap).

Source images are something similar to the one I've shown in my previous post.

Screenshot of my generated 3D is :


I'll post the database when I've uploaded it somewhere

Thanks for your time ;)

Christophe

[cbalthasart]

Here is the db file :

Database file

[matlabbe]

The link doesn't work...

[cbalthasart]

oups, link is already expired :(

This one should be valid for 7 days

Rtabmap database file

[matlabbe]

The format of the depth image seems not right. Here is the 3D cloud created from the single image (blue=closer, red=farther):



I tried with your sample images you posted here, the depth format is indeed wrong. The format is 16 bits in mm. 0mm=black, 65535mm=white. Closer stuff should appear black, while farther stuff should appear lighter in a standard PNG viewer. In database viewer, like in the screenshot above, depth image values are inverted (white=close, darker=far, black=invalid) for convenience just for visualization.

cheers,
Mathieu

[cbalthasart]

Thank you for your reply.

I've modified the depth image and now I can reproduce my horse.
Next step (for me) is about coordinates conversion between rtabmap and unity.

I'll post some screenshots of my results next week ;)

cheers,
Christophe