Develop sensor driver for OpenCV OAK-D to standalone RTAB-Map

[knoname]

Hi Matlabbe,
I'm interested in developing a camera driver for RTAB-Map (standalone) to interface to the OpenCV OAK-D camera.  The OAK-D has a proprietary interface but is open-source.  It has multiple features -- my intent is to start with something simple.  

If you haven't heard of the OAK-D, here is a link:  https://store.opencv.ai/products/oak-d

Today I can read the left & right cameras and the internally computed depth map in OpenCV (c/c++).  Even though the left and right cameras are technically monochrome, I could easily make an RGBD image from one of them and feed it to RTAB-Map.  

I've been looking in the cameras directory but most of the RGBD cameras have their own proprietary interfaces.  Do you have any suggestions of key places to look inside RTAB-Map or a good example I could modify?

I appreciate any help!

Thanks,
Bob.

[matlabbe]

Hi Bob,

The OAK-D interface looks also proprietary: https://github.com/luxonis/depthai-core/blob/main/include/depthai/device.hpp

You would have to make a CameraOAKD class. For an example, it depends if the images are received in callbacks or on request (e.g., device.capture()). You can look at the Camera abstract class, you would have to override those functions in subclass:

virtual bool init(const std::string & calibrationFolder = ".", const std::string & cameraName = "") = 0;
virtual bool isCalibrated() const = 0;
virtual std::string getSerial() const = 0;
virtual bool odomProvided() const { return false; }
virtual SensorData captureImage(CameraInfo * info = 0) = 0;

init: this is where we create the device, get camera intrinsics and start the device.
isCalibrated: return true if captured images are rectified.
getSerial: return serial of the device.
odomProvided: if the device provides odometry
captureImage: You have to set left/right + stereo model for stereo images in a rtabmap::SensorData. For RGB-D data, you have to set rgb image and depth image + camera model of the rgb camera. If odometry is provided, it should be set in optional Camera info argument (if not null).

cheers,
Mathieu