Illumination issues with RTAB map!!!

[kartik]

Hi,

I am using RTABmap for a while now. But, till date I was using it for mapping and navigation in indoor environments which have opaque walls and are not prone to external lighting conditions. Recently, I moved to a new lab and the whole lab is glass-walled which makes it completely prone to changes in the lighting conditions outside. The following are some observed problems:

1. Say, we try to map the given environment in the morning hours around 8:00 am, now when I test it in and around that time, like till 10:00 am, it works perfectly. I mean it localizes perfectly and all. But as it becomes noon the lighting conditions in the lab change and it's not able to localize as robust as before.

2. Let's say by some means I tried to make it localize, but when I try to navigate from point to point it often gives me an error saying that the robot has lost its pose. Then I manually go there and try to move it and try to make it localize again.

These are some of the issues I am facing and since most of the times I get this localization errors I feel the problem lies in the visual odometry part and not in the planner side. Please correct me if my hypothesis is wrong.

I am stuck with this problem for a while now, and I am thinking of applying some illumination normalization to rectify this but before doing anything I wanted to confirm that if this really is the problem I need to solve. Because I think there might have been cases like this before where visual SLAM fails without proper illumination etc.

I am using a setup which consists of KINECTV2 for mapping and I use an RPLidar for obstacle avoidance while navigating.

Any help in this regard would be greatly appreciated.

Best,
Kartik

[matlabbe]

Hi Kartik,

You may consider using wheel odometry if you have a robot to avoid the problem of lost visual odometry. Carefully calibrate your wheel odometry though. Using icp_odometry may be also a solution if the lidar can see the walls or furniture. However, as the walls are made of glass, it may be also difficult to use lidar for odometry. So I would start by replacing visual odometry by wheel odometry.

For localization, if rtabmap is subscribed to scan and the environment has enough geometry, rtabmap could localize with the lidar (assuming "RGBD/ProximityBySpace" is true). To get visual loop closure detection working in different illumination conditions, it could be possible to map the same environment on multiple sessions at different hours. Assuming rtabmap can localize on the map created 30 min or 1 hour before, it would merge all sessions together. That resulting map (with multiple sessions) could be used afterwards for localization at different hours.

cheers,
Mathieu

[kartik]

Hi Mathieu,

Thank you very much for the reply.

Coming to your first suggestion of using wheel odometry, I have already tried it and it works terribly with my system as the floor is way too skiddy (and we use rubber tiers)  and the robot is a wheelchair with a differential drive system.

For the localization, I will definitely check out the  "RGBD/ProximityBySpace" flag, as I did not try this before. \

I am still not sure about the idea you gave for the illumination part, sure I would try that but I still have some questions
1. How many such multisession maps do you think I would need ( assuming that I need to use the bot from 9:00AM to  7:00PM)
 2. I have tried the multi-session mapping before and I noticed that its a real memory intensive process and sometimes my computer hangs while RTABMAP tries to merge those huge maps. Is this common? if so is there any alternative for this method?
PS: The size of one session that I mapped was of 300MB
3. What if I try to add a piece of code for illumination rectification, something like this

is this possible? my reasoning behind this was I thought I can rectify such images(illumination prone images) and publish them to the RTABMAP node so that it can always see rectified and illumination invariant images.

Please let me know your opinion regarding this. Thanks for your patience.

[matlabbe]

Hi Kartik,

Indeed, you can give a try with exposure/illumination normalization. However, depending on the environment, having light coming from different directions can cause surfaces not looking the same even if you normalize the exposure/illumination. Shadows can be also different, as well as how contrast will appear. For example for a slightly reflective surface, it may look very different depending on where the light is coming from and what is the current point of view of the robot.

The idea behind doing multiple sessions is that the robot will "learn" in some way how lighting affects the surfaces during the day.

Creating multiple maps will require more memory and computation time over time (assuming memory management is disabled, which is the default). One strategy is to teleoperate the robot at different time to create a new database for each session (make sure to revisit always the same areas so that loop closures can be found between the sessions in the next step). Afterwards, we can use rtabmap-reprocess tool to merge all those maps offline into a single database:

$ rtabmap-reprocess "input1.db;input2.db;input3.db" "output.db"

We can also use rtabmap-detectMoreLoopClosures tool to find more constraints between all sessions to get a better optimized merged map:

$ rtabmap-detectMoreLoopClosures output.db

When you will launch the merged database with "Mem/IncrementalMemory=false" (localization mode), it should not use a lot of computation power to do the localization as the memory doesn't need to be updated.

To know how many sessions you need to do is a good question. In a project we mapped each ~15/20 minutes when the lighting was changing fast, like at around dawn or dusk (e.g. 7 am to 9:00 am, where 7 am is completely dark outside, and 9 am is sunny). In this example, we would have ~8 databases to merge. You may experiment doing a new session only each 30 min for example, and see how localization performance is affected.

If you don't need the 3D map, you can avoid saving the RGB and depth images to database, by setting "Mem/BinDataKept" to false, though you may only set this when creating the merged database for to be able to still debug individual sessions:

$ rtabmap-reprocess --Mem/BinDataKept false "input1.db;input2.db;input3.db" "output.db"

cheers,
Mathieu

[kartik]

Thank you so much for the reply. I tried to implement the solution that u proposed, and while doing so I bumped into some more issues which are as follows:

1. First, I tried multisession mapping
        This command doesn't work
         rtabmap-reprocess "map1.db; map2.db; map3.db" merged.db
         It returns an error saying......"Input database "map1.db; map2.db; map3.db" doesn't exist!", though it is a minute bug I think it is worth looking into this
         But, I followed your tutorial on multi-session mapping and was able to generate a multi-session map, Yet, there are
         some issues such as:
          i) after merging the maps I was not able to get any graph view(2d map) from the merged .db file,
              3D(left) and 2D(right) maps look something like this
             
             
              But in general, I find something like this
               

            can you please comment on this?

           ii) I wanted to save the database(.db) using the "Tools>DataRecorder" option, but it never saves anything, I dunno if this is actually the purpose of it but I felt like asking because it was saving only 30-50kB files
              whenever I try to save a database after multisession or preprocessing.
              It shows something like this:
             

              P.S: I know that when you choose the  "close database" option, it prompts for saving the current progress. But, isn't the "Data recorder" option meant for saving our work whenever required?
             

2. One more thing which might not be totally related to the illumination issue,
     So, when I finish one of my mapping sessions and launch Rviz for navigation part, I see this kind of unnatural visualization in which the 3D pointcloud (3D map) is a certain angle w.r.t to 2D map. It looks like this


     I dunno much if this is affecting my performance during navigation( I feel it is not) but it kind of looks strange when I try to see where the bot is moving because of the "half-submerged map" is this a problem related to TF ?


Thanks for your patience .
     

[matlabbe]

Hi,

1.i) Are you using rtabmap kinetic binaries? Merging multiple databases is supported by rtabmap-reprocess only when rtabmap is built from source (latest version). To use rtabmap desktop to merge the databases, follow the multi-session example for RTAB-Map Tango, in particular steps 3 and 5. In step 3, by loading parameters from your database, the 2d grid map should be created afterwards and seen in graph view.

1.ii) Data recorder is used for different purpose, to record images from a camera into database format. You should do "File->Close database" to save the current map shown in rtabmap.

2) When the whole map is tilted, it means the camera is tilted in reality but not in TF (or vice-versa). Open RVIZ and launch only your robot (without mapping), set global frame id to the base frame of your robot,  add the point cloud of the kinect and the cloud on the ground should be aligned with the x-y plane in RVIZ. If not, either adjust the camera to fit TF or modify TF to match the current camera position/orientation. Assuming the robot is moving always on the same flat ground, when the camera is correctly aligned with the ground, you can set Reg/Force3DoF to "true" (for rgbd_odometry and rtabmap nodes) so that over time, the errors in z, roll, pitch won't added (2D SLAM).

cheers,
Mathieu

[kartik]

Hi,

I seem to understand what you said. But, I already tried the multi-session mapping tutorial that you suggested. and this is the output:


it still shows nothing in the graph view.

Best,
Kartik

[kartik]

For the reference, here are my settings for multi-session mapping





Hope this helps.

Thanks in advance.

[matlabbe]

Go in Preferences, and make sure "Create local occupancy grid maps" is checked when building the map.


If this doesn't work, can you share the database through a file sharing service?

[kartik]

I am not able to expand the "RTAB-Map settings" section, and I see no more options than these:

[kartik]

Here is the link to the database folder

https://www.dropbox.com/sh/o12dxvlhnsubp2t/AABOpImpnsm2vwGRVIuDqsoDa?dl=0

[matlabbe]

Hi,

At the bottom left in Preferences dialog, click on "Advanced" to see all parameters.

I tried your database following step 3 an 5 of this tutorial on both kinetic binaries and latest rtabmap from source and it should show the 2d grid map.

Make sure that in the context menu (right-click) over the Graph View that the grid map is not hidden (the corresponding action should be labelled "Hide grid map", not "Show grid map").
 

[kartik]

Thank you so much. It worked on my old kinetic desktop version of rtabmap. Now I will try to compile the source version.
Will let you know how things work out with the idea that you gave for tackling illumination issues.

Cheers,
Kartik

[matlabbe]

For the tilting problem, I looked in more details the first database:

$ rtabmap-databaseViewer map1.db

1) On right we see a side view of a cloud (make sure to uncheck Odom Frame to see in base frame). I've drawn the red line on the points on the ground, normally it should be aligned with the grid. TF should be adjusted or the camera should be tilted.
2) The RGB and depth images are poorly registered. Make sure to do a good iai_kinect2 calibration. This will increase a lot the quality of the 3D map as well as the localization accuracy.


cheers,
Mathieu

[kartik]

Thanks again. I will definitely look into that.

One more minor question. how is the 3D point cloud that you showed smoother than mine? did you perform any post processing>

my cloud


the one which you just showed

[matlabbe]

I didn't do any post-processing. The clouds look similar though. The only thing I've set is the maximum depth of the clouds created for the map to 4 meters. See Preferences -> 3D Rendering -> "Maximum depth" under Map column to avoid far noisy points added to map. The background color is also modified (option is on right-click on Graph View).

cheers,
Mathieu