Missing Frames

[Montana]

Hi everyone, Just getting started playing around with the RTAB-Map standalone GUI and an Intel D435i camera on the RPi4. I built realsense 2.44 without and problems, and can run the realsense-viewer app to see everything from the sensor just fine. I built open-cv on the pi using the extra modules, notfree, etc flags just fine, along with g2o. I ran into some problems after building RTAB-map. First, for anyone who comes along and has issues trying to run the GUI with QT5 and gtk, make sure to change 'style=gtk2' with 'style=gtk3' in /etc/xdg/qt5ct/qt5ct.conf When I try to run mapping with the realsense 400 as RGB-D I run into a problem. No matter what I set resolutions, fps, or any other combination of options, after a few seconds the program stops and tells me:

[ERROR] (date - time) CameraRealSense2.cpp:1453::captureImage() Missing frames (received 1, needed=2)
[Warn] (date - time) CameraThread.cpp:189::mainLoop() no more images...

There is another pose here about what seems to be the same problem, but this seems to be a different reason, or that fix would be working here. There also is no delay between the stream stopping and this error, they seem to happen together. I can immediately unpause and it will start running again fine for another few seconds before repeating. It seems like using these cameras is popular, so I feel like I just have a setting wrong somewhere, or maybe missed an option when building (I built everything in release). Any help would be appreciated.

[Montana]

Doing some digging, it seems like the timestamps from frame.get_timestamp() on the rpi can be very wrong, and jump forward greatly. The syncer then gets confused when one frame jumps forward and the other was the old time.

DT on video frames (15hz target, units ms):
72.48
74.75
72.49
70.22
0.00    <== this frame is actually a repeat of the previous
27178.48     <== big jump forward, then continues on
70.18
70.18
70.18
70.18

A time stamp that is more consistent, but less accurate, is frame.get_frame_metadata(RS2_FRAME_METADATA_TIME_OF_ARRIVAL), which is a system time from after the usb message was received from the device. See here.  Other options would be to look for these occurrences and drop them and wait for the next frames to process.

[matlabbe]

Hi,

you could try if playing with enabling or disabling RS2_OPTION_GLOBAL_TIME_ENABLED could help:
https://github.com/introlab/rtabmap/blob/6449302318eef03d52974e5ba6d4165ced498d5c/corelib/src/camera/CameraRealSense2.cpp#L1060-L1065

The global time sync option is available in Preferences->Source->RealSense2.

Check also if the camera has the latest firmware.

cheers,
Mathieu

[Montana]

Thanks for the reply. I've collected some data directly from the sensors with nothing else running to help shed some light on the issue.

For all of these the columns are the DELTAS (time between frames) for color+depth of the following functions, in their native units:

frame.get_timestamp(), frame.get_frame_metadata(RS2_FRAME_METADATA_FRAME_TIMESTAMP), frame.get_frame_metadata(RS2_FRAME_METADATA_TIME_OF_ARRIVAL)

First dataset is from only the D435I with GLOBAL_TIME_ENABLED=0 (off)

69.89, 66720, 66
67.77, 66720, 67
65.66, 66720, 67
5,918.18, 66720, 67
67.27, 66720, 66
67.28, 66720, 67
69.37, 66720, 68
65.17, 66720, 66
69.38, 66720, 67
67.27, 66720, 67
65.17, 66720, 66
67.28, 66720, 67
69.37, 66720, 66
67.28, 66720, 67
67.27, 66720, 67
67.27, 66720, 66
67.28, 66720, 67
69.37, 66720, 67
2,595.55, 66720, 67
67.06, 66720, 66

Second dataset is from only the D435I with GLOBAL_TIME_ENABLED=1 (on)

68.67, 66720, 67
66.60, 66720, 66
66.59, 66720, 67
66.59, 66720, 67
100.86, 66720, 67
64.51, 66720, 66
68.68, 66720, 67
66.59, 66720, 67
64.51, 66720, 66
68.68, 66720, 67
66.59, 66720, 67
66.59, 66720, 67
64.52, 66720, 66
68.67, 66720, 67
68.68, 66720, 67
66.59, 66720, 67
66.59, 66720, 66
66.60, 66720, 67
66.59, 66720, 67
86.49, 66720, 66

Third dataset is from both the D435I and T265 with GLOBAL_TIME_ENABLED=1 (on)

66.79, 66720, 67
485.06, 66720, 66
66.76, 66720, 67
68.84, 66720, 67
66.76, 66720, 67
66.76, 66720, 66
66.75, 66720, 67
66.76, 66720, 67
68.85, 66720, 67
66.75, 66720, 66
64.68, 66720, 67
68.84, 66720, 67
66.76, 66720, 66
68.84, 66720, 67
66.76, 66720, 67
64.67, 66720, 66
428.70, 66720, 67
66.73, 66720, 67
66.72, 66720, 66

As you can see the first column, representing frame.get_timestamp() and the supposed sensor timestamp for when the sample was taken, is entirely unreliable. I went ahead and installed realsense and rtabmap on a windows PC, and the same issue seems to apply there as well. Using only an D435I and global timestamps there are few, if any, instances of the missing frame error. Using the dual mode with the T265 plugged in and they seem to crop up often. I feel like someone else should have seen this behavior, since these are pretty popular sensors, but maybe there is some other option or influencing factor. Meanwhile I have been rewriting the realsense2 camera driver to account for this and some other issues, but it may not be suitable to give back to the project when I am done.