Deblurring/Motion Correction for Handheld Low-light RGB Images

[Alan]

Hi everyone. Sorry if this topic has been covered already, but I'd like to offer a suggestion for improving the quality of the RGB images recorded in certain special cases.

Our particular application involves hand-held mapping using a Tango device in low-light settings. We have been able to achieve impressive point cloud results by using a gymbal for stabilization, however obtaining good RGB images for rastering is proving to be difficult, even when we use bright lights.

One possible solution could be data collection strategy that uses faster shutter times on the RGB camera and post-processing for motion correction and signal recovery. Essentially, the device would be recording movie with each frame having a much faster shutter speed than what is currently used. This would give frames that are darker with more noise, but importantly less motion per frame. In post processing, blocks of ~20 frames can then be stacked and motion corrected (aligned), giving a final image that has increased brightness, reduced noise and overall less motion blur.

There are academically developed (open source?) software packages that accomplish this with very poor signal-to-noise images using information present in the images only. Maybe in the case of Tango devices, the depth sensor data could be used as a constraint.

Again, my understanding of the software implementation in RTAB-Map isn't very complete so I apologize if this has already been discussed. I wish I was a little better with coding so I could give an implementation of this a try myself.

-A

[matlabbe]

Hi Alan,

This is an interesting solution, but I would see this done upstream to rtabmap. For example, with rtabmap updated at 1 Hz, a module could accumulate 30 frames (in case of 30 Hz camera) during that second, apply motion correction and signal recovery, then send the result at 1 Hz to rtabmap. The difficulty is to correctly align the images together using the motion estimated by tango during that time. Not sure about the computation power required to do this though.

There are academically developed (open source?) software packages that accomplish this with very poor signal-to-noise images using information present in the images only.

Can you refer some links (papers or projets)?

cheers,
Mathieu

[Alan]

The only examples I'm aware of are oriented towards electron microscopy and might require a little bit of unpacking to re-purpose towards colored images. Here is one example: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3684049/pdf/nihms467651.pdf
More recent iterations of these algorithms split the image into sections (which wouldn't be needed in this case).

Unfortunately I think the process is fairly CPU intensive, but perhaps using information from the depth sensor could reduce the computational load. In that case I would think the depth sensor would have to be recording at a similar frequency as the camera in order to really help.