Standalone RTAB -map for Realsense D435 (Intel® RealSense™ SDK 2.0 )

[rasulnrasul]

Hi Mathieu,

I want to use standalone RTAB-map with realsense D435 ( without ros). From the github issue discussion here, I understood that D435 is not supported for standalone rtab-map  as of now, as the realsense sdk is changed completely i.e librealsense2.

Can I know, your plans on integrating Intel® RealSense™ SDK 2.0.

Thanks,
Nayab

[matlabbe]

Hi Nayab,

We are currently waiting for a camera that we bought weeks ago, not sure when we will receive it (they are back ordered). When we will receive it, it won't be too long (a day?) to support it.

cheers,
Mathieu

[Eric Schleicher]

Are you planning on mapping, or just using the realsense for navigation?  if the prior, i can save you some time, effort and frustration.  I've used the realsense with RTABMAP via ros, and thorugh no fault of RTABMAP, the quality of the captured data is really quite poor.  

essentially the 400 series camera can see much further than the 200/300 series, but the noise and latency (in the driver stack) are so bad, that the camera is effecively useless for anything beyone 1.5m.  and even then, you need some type of more sophistication reconstruction algorithm (like kinfu, or other TDSF) to smoothout the noise.

If you want to see a comparison of the streams between the Kinect2 and the realasense 435 here is a video.

https://youtu.be/JPxsNXNsciA

Probably not what you want to hear, but hopefully useful...  many are pretty dissapointed with these cameras on a wholesale basis.

-Eric

[rasulnrasul]

Hi Eric,

Thanks for the information. I am planning to use D435 for mapping and for navigation. Previously I have tried mapping with ZR300 and it was noisy. As of know, I have just started using (D435 + RTABmap library integrated into our system) and want to check how good the final 3D map will be.


Thanks,
Nayab.

[gxcl2018]

hello- I discovered rtab-map when the release of tango applications on my lenovo- I thank you for the quality of your developments- I received after a few months my D435 realsense, do you think I'll soon be able to use it with rtab-map? thank you in advance for your reply

[rmapbot]

matlabbe wrote

Hi Nayab,

We are currently waiting for a camera that we bought weeks ago, not sure when we will receive it (they are back ordered). When we will receive it, it won't be too long (a day?) to support it.

cheers,
Mathieu

Hello Mathieu,

I have purchased a D415 sensor and I am excited to try it with RTABMAP. I have begun to build RTABMAP with realsense sdk2.0. I believe its just a matter of updating the CameraRealSense class to use the new rs2 api.

By any chance, do you already have this CameraRealSense2 class implemented that I could pull or copy into my build?

I would be willing to send you my sensor if it would help you to integrate the rs2 sdk.

We are going to continue to look at integrating rs2, but please let us know if you have already done the work.

Thank you!
-Matthew

[j]

Hi Mathieu,

I am also interested in the Realsense D435 integration. I currently have it communicating over ROS with the rtabmap-ros library, however the noise of the camera make the composite point clouds almost unusable. Do you have any advice on how to reduce the noise of the RGBD input?

Thank you!
-Johan

[matlabbe]

Hi Matthew,

I received the D435 yesterday, RealSense2 sdk is now integrated to RTAB-Map standalone (see master branch). So far, I just tested on Ubuntu and with a D435, not sure about the D415 but you can try if the D435 driver can work with it.

cheers,
Mathieu

[matlabbe]

Hi Johan,

You may decrease the maximum depth of the clouds added to map to keep only closer points that are more accurate. In rtabmapviz, see Preferences->3D Rendering and change max depth under Map column. There is a similar parameter for RVIZ rtabmap_ros/MapCloud plugin.

cheers,
Mathieu

[amiltonwong]

Hi, Mathieu,

I had tested rtabmap with my new D415 (not D435). After building and installing the most updated librealsense sdk and rtabmap. I got the following running error when using D415:

root@milton-ThinkPad-T450:~# rtabmap
[ WARN] (2018-06-25 19:27:01.682) Parameters.cpp:838::readINI() Section "Core" in /root/.rtabmap/rtabmap.ini doesn't exist... Ignore this warning if the ini file does not exist yet. The ini file will be automatically created when rtabmap will close.
[ERROR] (2018-06-25 19:27:23.368) CameraRGBD.cpp:3609::captureImage() An error has occurred during frame callback: Frame did not arrive in time!
[ WARN] (2018-06-25 19:27:23.368) CameraThread.cpp:146::mainLoop() no more images...


"No more images..." error was encountered. Please see also the attached screenshot pic "https://www.dropbox.com/s/h85s2o01xxesrdu/rtabmap_d415_error.png?dl=0"


It seems the synchronization problem in reading the input stream, right?
Could you suggest me how to fix this error?

THX!
Milton

[cbalthasart]

Hello,

I have the same error with my D415 : Frame did not arrive in time

The D415 works fine with the realsense2 drivers (using rs-capture, depth and rgb are shown).

I've already used the realsense2 driver for the D415 to retrieve RGB, Depth and Infrared and it works fine too (I'm using rsPipeline->wait_for_frames() to retrieve a rs2::frameset but I didn't use a timeout)

Chris

[cbalthasart]

I've tried to find the problem. I've put some debug trace in the function captureImage of the realsense2 class.
The exception comes from the wait_for_frames() in the while loop because there is only one image available (instead of 2) :

[ WARN] (2018-06-25 17:34:35.129) CameraRGBD.cpp:3537::captureImage() Trace 1.
[ WARN] (2018-06-25 17:34:35.130) CameraRGBD.cpp:3539::captureImage() Trace 2.
[ WARN] (2018-06-25 17:34:35.130) CameraRGBD.cpp:3541::captureImage() Nb of frames in frameset : 2.
[ WARN] (2018-06-25 17:34:35.130) CameraRGBD.cpp:3547::captureImage() Trace 3.
[ WARN] (2018-06-25 17:34:35.130) CameraRGBD.cpp:3556::captureImage() Trace 4.
[ WARN] (2018-06-25 17:34:35.130) CameraRGBD.cpp:3572::captureImage() Trace 5.
[ WARN] (2018-06-25 17:34:35.130) CameraRGBD.cpp:3576::captureImage() Trace 6.
[ WARN] (2018-06-25 17:34:35.130) CameraRGBD.cpp:3578::captureImage() Trace 7.
[ WARN] (2018-06-25 17:34:35.130) CameraRGBD.cpp:3587::captureImage() Before alignFrame.
[ WARN] (2018-06-25 17:34:35.208) CameraRGBD.cpp:3591::captureImage() After alignFrame.
[ WARN] (2018-06-25 17:34:35.209) CameraRGBD.cpp:3605::captureImage() Trace 8.
[ WARN] (2018-06-25 17:34:35.209) CameraRGBD.cpp:3607::captureImage() Trace 9.
[ WARN] (2018-06-25 17:34:35.209) CameraRGBD.cpp:3537::captureImage() Trace 1.
[ WARN] (2018-06-25 17:34:35.209) CameraRGBD.cpp:3539::captureImage() Trace 2.
[ WARN] (2018-06-25 17:34:35.209) CameraRGBD.cpp:3541::captureImage() Nb of frames in frameset : 1.

SensorData CameraRealSense2::captureImage(CameraInfo * info)
{
	SensorData data;
#ifdef RTABMAP_REALSENSE2

	try{
        UWARN("Trace 1.");
        auto frameset = syncer_.wait_for_frames(5000);
        UWARN("Trace 2.");
        UTimer timer;
        UWARN("Nb of frames in frameset : %d.", frameset.size());
		while (frameset.size() != 2 && timer.elapsed() < 2.0)
		{
			// maybe there is a latency with the USB, try again in 10 ms (for the next 2 seconds)
			frameset = syncer_.wait_for_frames(10);
		}
        UWARN("Trace 3.");
        if (frameset.size() == 2)
		{
...

Chris

[matlabbe]

Hi,

I did reproduce the problem with my D435 when plugging it to a USB2 port instead of USB3 port. I increased the wait time in this commit to fix it with USB2: https://github.com/introlab/rtabmap/commit/62a64cd15631dbd68ec46ac28663cdd0e6c49021

Well, I didn't try for a long period, let me know if it still happen.

cheers,
Mathieu