Bunlder export: what poses ?

[Pierre]

Hi,

I noticed that exporting in Bundler format from the GUI app always leads to different poses than when I export clouds in the usual way. Also they don't just differ from a rigid transformation.

My goal was to have a bundler export of my project, as well as a nice cloud with normals, compatible with the bundler export. I couldn't manage to do so.

Also it doesn't seem to just be a consequence of the additional SBA done before the bundler export: I tried exporting clouds before and after the bundler export, + after doing additional SBAs in case the bundler ones are not kept after exporting. Nothing could match the bundler poses.

Thanks for your help
Best
Pierre

[matlabbe]

Hi Pierre,

Are the poses very far from each other?

Is "Export 3D points" checked? If so an optimization will be always redone, which can give slightly different results each time.

Based on this : https://github.com/introlab/rtabmap/wiki/Export-Raster-Layers-to-MeshLab , you may want to keep "Export 3D points" unchecked.

[Pierre]

Hi Mathieu,

Thank you for your answer. It could be that I'm doing something wrong although I'm more or less doing exactly what the tutorial suggests.

Here is a screen of the cloud (3d points) from the cameras.out, and the cloud exported from RTABMap.

What I did is, exactly,
- open the db and chose global optimized
- export bundler without closing loops, and chosing 20 iterations of sba (and saying yes to 3d points because I want them) (---> I misread your message, that's why there is a slight difference everytime...)
- export clouds



The sba iterations done for the bundler export are actually kept and applied for the later cloud export aren't they ?

EDIT: I absolutely misread your last post, sorry. So the sba iterations are not kept after the bundler export then... Which means there is currently no way to export 3D points + a compatible mesh ?

[matlabbe]

Pierre wrote

The sba iterations done for the bundler export are actually kept and applied for the later cloud export aren't they ?

That is a good idea, currently the optimized poses are not applied back to current poses loaded in the main window. That could fix your issue if we update them back, then Export Clouds will use the same poses. I created an issue here: https://github.com/introlab/rtabmap/issues/1483

EDIT: https://github.com/introlab/rtabmap/issues/1483 has been fixed.

[Pierre]

Amazing Mathieu ! This was fast :)
Thanks a lot

[Pierre]

Hi Mathieu,

I  have come to realise that in many (most) of my databases, the bundle adjustments make the camera poses way worse than they were before. This is probably due to the fact that many of the rgb images have sparse and unconsistant lighting (I use a frontlight).

I can provide databases if you're interested. In practice, if I export clouds before any BA, I can do a "high deph" Poisson reconstruction which recovers a good amount of details. After a single iteration of BA, a Poisson reconstruction at the same depth produces triple walls and ceiling artifacts due to unwanted shifts in camera poses. I'm showing an example at the end of this post, although it's not so easy to see with just one screenshot.

Hence I was wondering if it would make sense to allow a Bundler export _with 3D points_ with 0 iterations of BA (right now at least one is required).

Thank you for your help,
Best,
Pierre

Example: backface culled mesh of a cave-in seen from outside. On the first mesh one can see inside the cave-in. The second mesh is after BA, same Poisson depth, but one cannot see inside even if backface culling is on, because of the "double/tripple wall" artifact.

[matlabbe]

I could be interested to see the database, in particular to reproduce the bad BA optimization.

It is a good idea to let use 0 iteration for no optimization, I updated it in this commit.

cheers,
Mathieu

[Pierre]

Hi Mathieu,

Thanks a lot for the super quick commit ! It's very helpful for me.

Here is the database: https://drive.google.com/file/d/1mf6UlY9VnzPPzDLvGX1bD-RTOLnhRlrI/view?usp=sharing

This is just one example, but I found that the issue is the same with a lot of similar databases that I have.

Best,
Pierre

EDIT [bug]: now we can indeed select "0" for the number or BAs. But even when selecting 0, one iteration of BA is done.

[matlabbe]

Hi Pierre,

I verified, and there should not be any BA when iteration=0. Features still need to be re-matched, so it is why it still take a while to export. I however added more logs to confirm this and fixed an issue where SuperGlue was not used when BA iteration >0.

In this database, I don't see a large difference between before and after BA. Here before and after BA respectively, with a side-view of some point clouds that should represent the same surface:




The problem here is not BA or not BA (results look very similar), it is that the depth is poorly estimated from the raw input. When assembling a mesh, having overlapping surfaces like this would produce unpredictable effect depending on Poisson depth value (I would recommend to set it the lowest as possible, like between 6 and 8). Here is a comparison between two frames. We can see in one frame the rocks formation produces a bump in the depth, while the same rocks in the other frame are completely flat.



Which iPhone or iPad model are you using? Which iOS version? In RTAB-Map's App, what is the value set in Depth Confidence parameter? These results are concerning if ARKit relying less and less on LiDAR data to compute depth values. For me, this kind of depth looks more like coming from a not so accurate monocular camera depth estimation approach than LiDAR-based depth (unless the LiDAR data is super wrong).



Regards,
Mathieu

[Pierre]

Hi Mathieu,

Thank you for your answer.
I’m sorry I probably misunderstood the log of the Bundler export, I could see « BA » written at each line so I got mistaken. I’ll recompile rtabmap in case it’s an issue on my side.

Maybe it was a coincidence that Poisson reconstruction performs better on clouds before BA, then, seeing the tests you made. Although I really got consistently better results with several databases of that kind, always better before BA. The only reason I can think of that could explain this is if 3D points were calculated using depth map values outside of masked regions instead of SfM, in which case their positions would be biased by poor depth maps and BA would possibly not be able to make the situation better — although I hope it is not the case ?

For the depth confidence, I am using medium. I have to, because scanning this kind of environments with high confidence is close to impossible: depth maps are extremely sparse with very little information. My iOS version is 18.0.1. I do agree that ARKit depth maps are often poor. I’m using an iPhone 15 pro which if I understood correctly has fewer dots but better algorithms… although not good enough.

An option that would be helpful for iOS scans is to provide the depth confidence map to the user instead of having to chose, although we already talked about that and I know it would need some more thoughts (and it is already existing feature request). With that information, a weighted Poisson reconstruction could be performed and I would be very interested in seeing some results of that. Some confidence-biased noise removal could also be considered at the point cloud level (not asking that any of this get implemented in rtab, I’d be happy to work with the raw data myself !)

Is it possible to add an additional field only for databases coming from ARKit, or the structure has to be fully uniform ?

Thanks a lot for all the support,
Best
Pierre

[matlabbe]

Hi Pierre,

That hurts my soul seeing such scanning effort and not able to change the confidence level after the scan has been taken. With the depth confidence, at least we could recover the data we want in post-processing. I'll take a look this week at how to integrate depth confidence in RTAB-Map in short-term, without doing the more general approach discussed in that issue. Other related post.

For why Poisson looked better without BA, if the Poisson depth is not explicitly set, it could have not using the same depth. Lower Poisson depth can better average overlapping surfaces into just one, though we lose geometric details. The BA could also made it worst, the bad depth shown in my previous post may have created not very accurate loop closures, thus adding more errors than reducing them.

For the Poisson-depth confidence aware reconstruction, I could see an algorithm doing similar to this function that would keep low confidence points on a surface only if there are no high confidence points nearby.

cheers,
Mathieu

[Pierre]

Hi Mathieu,

I thank you so much for giving credit to my issues once again :)
I indeed have huge scanning projects, of sizes that are considerably larger than the example database I sent. Having the possibility of recording confidence will definitly change my range of possibilities.

Poisson depth was the same for the two reconstructions, usually I only save the cloud with RTAB and then reconstruct on the side with Meshlab. But you seem to imply in your message that ARKit depth maps are used for PnP and 3D points, in which case it makes sense that BA can make things worse. Out of curiosity, is there a way to ask that 3D points are only calculated with SfM ? I know this is already implemented for masked regions with DepthAsMark.

matlabbe wrote

For the Poisson-depth confidence aware reconstruction, I could see an algorithm doing similar to this function that would keep low confidence points on a surface only if there are no high confidence points nearby.

That sounds good. I had a more "basic" idea in mind, which is to introduce weights in the Poisson equation itself. Or, maybe intuitively clearer, do a tsdf reconstruction and bias the tsdf weights with the depth quality. I think depth quality can fit well with these implicit/volumetric reconstruction algorithms.

Thanks again for your help,
Best

Pierre

[matlabbe]

Hi Pierre,

We recompute the visual features for loop closure detection, meaning we re-extract 2D keypoints and descriptors, then use depth image to estimate their 3D position. If the depth image has mostly null values, then BA done in rtabmap won't work great. We don't use the 3d visual features tracked by ARKit. Note also that "Mem/DephAsMask" may not always work (a warning should be shown if not), it requires that RBG resolution should be a multiple of depth resolution, if not, the depth is ignored and features are extracted whether there is depth or not. Features without depth are ignored during BA. That is a reason why using low depth confidence (dense depth) may be useful for features extraction and BA.

FYI, version 0.22.0 of RTAB-Map iOS is currently in review on appstore, it should be released tomorrow or early this week. In that version, we keep both the original ARKit's depth and confidence images in the database, so it is now possible to change the confidence level after scanning and re-export. At least until we have smarter reconstruction algorithms supporting the depth confidence, you will have already the data in your next scans.

cheers,
Mathieu

[Pierre]

Hi Matthieu,

Thanks a lot for adding that feature! I can now carry on scanning in total serenity, knowing everything is safely stored.

Regarding 3D points, I think I may not have phrased my question clearly enough. If I understand correctly, it’s not necessary to use depth from the depth image to calculate 3D points—since if DepthAsMask is off and StereoFromMotion is on, depth can be estimated via SfM in those areas.

So what I’m wondering is: is there an option that allows the user to always favor the SfM-based depth estimation instead of relying on the depth map? With an ARKit-based dataset, this would effectively be an ARKit-odometry-guided SfM, ignoring potentially poor depth maps. Sorry for the confusion earlier.
EDIT: I think I found it - Vis/EstimationType = "2".

One last question/suggestion—setting aside the SfM topic: when depth confidence is available, could it be used as a form of covariance input for PnP? For example, using weighted RANSAC PnP and potentially adapting the PnP covariance based on the depth confidence of the inlier set.

Thanks
Regards

Pierre

[matlabbe]

Hi Pierre,

If DepthAsMask is off and StereoFromMotion is on, features are extracted anywhere in the image even if there is no depth, then for those without depth, their 3D position (e.g., depth) is computed by triangulating them from their correspondence in the previous image. There are some caveats on why the depth computed like this would be almost worst than using the depth image.
1. If there is not a lot of overlap between the consecutive images, no depth could be computed, which is generally the case with RTAB-Map set at 1 Hz update rate by default.
2. The 3D position of the feature is estimated only by two frames, so if there is not a lot of translation, the 3D estimation will be generally poor.
3. Even if depth of ARKit can be poorly estimated, I think it would still be always be as "good" or even better than StereoFromMotion, because ARKit can estimate depth on multiple consecutive frames (more overlap). For example, we record the 3D features estimated by ARKit into the LaserScan data for each node, you can see how "good" or "bad" ARKit's 3d feature estimation is (by not using depth) using rtabmap-databaseViewer.

For Global Bundle Adjustment post-optimization, I would expect that depth computed from StereoFromMotion or "low confidence" ARKit depth would give similar result. Currently, we don't track confidence of the 3D position for each visual feature. With 3D feature confidence, we could tune "g2o/PixelVariance" parameter and increase it for features we know that were estimated by poor depth confidence or StereoFromMotion. By giving more pixel variance, is means the depth can change more during BA for these features.

Vis/EstimationType = "2" is kinda experimental, it is completely ignoring depth during pose estimation, but uses depth of features to adjust the scale of the transformation computed. PnP (Vis/EstimationType=1) gives more stable results.

I am not sure how confidence could be used in PnP, though it could be in BA as explained above. When Vis/BundleAdjustment=1 (default), a local BA is always done after PnP between the two frames. The weighted optimization from BA could be done there using features' confidence.

cheers,
Mathieu

[Pierre]

Hi Mathieu,

I see, thank you for the explanations. Introducing weights in the BAs sounds very nice.
For PnP I was thinking of the following way of using weights: when using RANSAC for the estimation of the transformation, change the way of selecting the winning sample set at each iteration, i.e. instead of selecting the transformation that has the largest inlier set, chose the one that has the best "score", where "score" is typically the sum of the weights of each inlier.

Regards
Pierre

[matlabbe]

Hi Pierre,

Thank you for the details. I added an issue to not lose track of this.

cheers,
Mathieu