Localize (stereo-to-stereo) against map Signature with non-map Signature

[tfs]

Hi there

I have a usage question, and before really digging in I thought I'd ask here as the answer may be obvious. I am building a map using the Memory interface,

memory->update(data, ...)

Now, I have a function that receives data from the same (or same kind of) camera, and I want to register that data against a location in the map, without updating the map:

Signature newFrame(newData);
Signature * mapFrame = const_cast<Signature*>memory->getSignature(id); //A bit ugly

Transform result = memory->computeTransform(*mapFrame, newFrame, ...);

The result is that while there are many keypoints detected in each frame (~900), the debug logs report zero matches between them. If I use RegistrationVis to compute newFrame1-to-newFrame2 or Memory to get mapFrame1-to-mapFrame2, it works great. But not newFrameA-to-mapFrameB.

[DEBUG] (2018-07-09 19:17:51.408) util3d_motion_estimation.cpp:98::estimateMotion3DTo2D() words3A=917 words2B=916 matches=0 words3B=916 guess=xyz=0.000000,0.000000,0.000000 rpy=0.000000,-0.000000,0.000000
...
[ WARN] [1531178271.409372240, 1528391239.075265943]: Registration failed: "Not enough inliers 0/10 (matches=0) between 5 and 0"

Is there a better way to do this?

Thanks

[matlabbe]

Hi,

I would go directly with RegistrationVis, through computeTransformation() interface:

Transform computeTransformation(
			const SensorData & from,
			const SensorData & to,
			Transform guess = Transform::getIdentity(),
			RegistrationInfo * info = 0) const;

Can you show the whole debug output of RegistrationVis? You may set guess to null (Transform()) so that global matching is computed between the two images.

cheers,
Mathieu

[tfs]

Thanks Mathieu

It works using the interface you suggested. Playing around with it a bit more, the issue only happens if I send a null or identity transform as the guess using:

registrationViz_->computeTransform(*mapFrame, Signature(newData), Transform(), &regInfo);

The Debug log of the above is pasted below. If I supply a non-null/identity guess it seems to work OK. It also seems like the number of matches is also dependent on how good the guess is?

Best

Terry

[DEBUG] (2018-07-16 11:42:56.354) RegistrationVis.cpp:196::computeTransformationImpl() Vis/MinInliers=10
[DEBUG] (2018-07-16 11:42:56.354) RegistrationVis.cpp:197::computeTransformationImpl() Vis/InlierDistance=0.100000
[DEBUG] (2018-07-16 11:42:56.354) RegistrationVis.cpp:198::computeTransformationImpl() Vis/Iterations=300
[DEBUG] (2018-07-16 11:42:56.354) RegistrationVis.cpp:199::computeTransformationImpl() Vis/EstimationType=1
[DEBUG] (2018-07-16 11:42:56.354) RegistrationVis.cpp:200::computeTransformationImpl() Vis/ForwardEstOnly=1
[DEBUG] (2018-07-16 11:42:56.354) RegistrationVis.cpp:201::computeTransformationImpl() Vis/EpipolarGeometryVar=0.020000
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:202::computeTransformationImpl() Vis/PnPReprojError=2.000000
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:203::computeTransformationImpl() Vis/PnPFlags=0
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:204::computeTransformationImpl() Vis/CorType=0
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:205::computeTransformationImpl() Vis/CorFlowWinSize=16
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:206::computeTransformationImpl() Vis/CorFlowIterations=30
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:207::computeTransformationImpl() Vis/CorFlowEps=0.010000
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:208::computeTransformationImpl() Vis/CorFlowMaxLevel=3
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:209::computeTransformationImpl() guess=xyz=[null] rpy=[null]
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:220::computeTransformationImpl() Input(1): from=858 words, 858 3D words, 858 words descriptors,  0 kpts, 0 kpts3D, 0 descriptors, image=1280x720
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:231::computeTransformationImpl() Input(0): to=0 words, 0 3D words, 0 words descriptors, 0 kpts, 0 kpts3D, 0 descriptors, image=1280x720
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:251::computeTransformationImpl() 
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:332::computeTransformationImpl() IDs are not unique, IDs will be regenerated!
[DEBUG] (2018-07-16 11:42:56.355) RegistrationVis.cpp:487::computeTransformationImpl() 
[DEBUG] (2018-07-16 11:42:56.355) util2d.cpp:1156::computeRoi() roi ratios = 0.000000, 0.000000, 0.000000, 0.000000
[DEBUG] (2018-07-16 11:42:56.355) util2d.cpp:1157::computeRoi() roi = 0, 0, 1280, 720
[DEBUG] (2018-07-16 11:42:56.355) util2d.cpp:1184::computeRoi() roi = 0, 0, 1280, 720
[DEBUG] (2018-07-16 11:42:56.373) Features2d.cpp:575::generateKeypoints() Keypoints extraction time = 0.017982 s, keypoints extracted = 1000 (mask empty=1)
[DEBUG] (2018-07-16 11:42:56.373) RegistrationVis.cpp:527::computeTransformationImpl() kptsFrom=858
[DEBUG] (2018-07-16 11:42:56.373) RegistrationVis.cpp:528::computeTransformationImpl() kptsTo=1000
[DEBUG] (2018-07-16 11:42:56.377) Features2d.cpp:607::generateDescriptors() Descriptors extracted = 904, remaining kpts=904
[DEBUG] (2018-07-16 11:42:56.377) Features2d.cpp:607::generateDescriptors() Descriptors extracted = 904, remaining kpts=904
[DEBUG] (2018-07-16 11:42:56.377) Stereo.cpp:116::computeCorrespondences() util2d::calcOpticalFlowPyrLKStereo() begin
[DEBUG] (2018-07-16 11:42:56.384) Stereo.cpp:129::computeCorrespondences() util2d::calcOpticalFlowPyrLKStereo() end
[DEBUG] (2018-07-16 11:42:56.384) Stereo.cpp:149::computeCorrespondences() total=904 countFlowRejected=5 countDisparityRejected=80
[DEBUG] (2018-07-16 11:42:56.384) RegistrationVis.cpp:727::computeTransformationImpl() descriptorsFrom=858
[DEBUG] (2018-07-16 11:42:56.384) RegistrationVis.cpp:728::computeTransformationImpl() descriptorsTo=904
[DEBUG] (2018-07-16 11:42:56.384) RegistrationVis.cpp:1107::computeTransformationImpl() 
[DEBUG] (2018-07-16 11:42:56.384) VWDictionary.cpp:309::update() 
[DEBUG] (2018-07-16 11:42:56.384) VWDictionary.cpp:509::update() Dictionary updated! (size=0 added=0 removed=0)
[DEBUG] (2018-07-16 11:42:56.384) VWDictionary.cpp:517::update() 
[DEBUG] (2018-07-16 11:42:56.384) VWDictionary.cpp:309::update() 
[DEBUG] (2018-07-16 11:42:56.384) VWDictionary.cpp:324::update() Incremental FLANN: Removing 0 words...
[DEBUG] (2018-07-16 11:42:56.384) VWDictionary.cpp:333::update() Incremental FLANN: Removing 0 words... done!
[DEBUG] (2018-07-16 11:42:56.384) VWDictionary.cpp:337::update() Incremental FLANN: Inserting 858 words...
[DEBUG] (2018-07-16 11:42:56.384) VWDictionary.cpp:364::update() Building FLANN index...
[DEBUG] (2018-07-16 11:42:56.384) VWDictionary.cpp:382::update() Building FLANN index... done!
[DEBUG] (2018-07-16 11:42:56.384) FlannIndex.cpp:347::addPoints() Rebuilding FLANN index: 1 -> 3
[DEBUG] (2018-07-16 11:42:56.384) FlannIndex.cpp:347::addPoints() Rebuilding FLANN index: 3 -> 7
[DEBUG] (2018-07-16 11:42:56.384) FlannIndex.cpp:347::addPoints() Rebuilding FLANN index: 7 -> 15
[DEBUG] (2018-07-16 11:42:56.385) FlannIndex.cpp:347::addPoints() Rebuilding FLANN index: 15 -> 31
[DEBUG] (2018-07-16 11:42:56.385) FlannIndex.cpp:347::addPoints() Rebuilding FLANN index: 31 -> 63
[DEBUG] (2018-07-16 11:42:56.385) FlannIndex.cpp:347::addPoints() Rebuilding FLANN index: 63 -> 127
[DEBUG] (2018-07-16 11:42:56.385) FlannIndex.cpp:347::addPoints() Rebuilding FLANN index: 127 -> 255
[DEBUG] (2018-07-16 11:42:56.386) FlannIndex.cpp:347::addPoints() Rebuilding FLANN index: 255 -> 511
[DEBUG] (2018-07-16 11:42:56.388) VWDictionary.cpp:396::update() Incremental FLANN: Inserting 858 words... done!
[DEBUG] (2018-07-16 11:42:56.388) VWDictionary.cpp:509::update() Dictionary updated! (size=0 added=858 removed=0)
[DEBUG] (2018-07-16 11:42:56.388) VWDictionary.cpp:517::update() 
[DEBUG] (2018-07-16 11:42:56.388) VWDictionary.cpp:595::addNewWords() id=2 descriptors=904
[DEBUG] (2018-07-16 11:42:56.388) VWDictionary.cpp:688::addNewWords() newPts.total()=904 
[DEBUG] (2018-07-16 11:42:56.394) VWDictionary.cpp:753::addNewWords() Time to find nn = 0.005758 s
[DEBUG] (2018-07-16 11:42:56.395) VWDictionary.cpp:874::addNewWords() naive search and add ref/words time = 0.000635 s
[DEBUG] (2018-07-16 11:42:56.395) VWDictionary.cpp:876::addNewWords() 904 new words added...
[DEBUG] (2018-07-16 11:42:56.395) VWDictionary.cpp:878::addNewWords() 0 duplicated words added (from current image = 0)...
[DEBUG] (2018-07-16 11:42:56.395) VWDictionary.cpp:879::addNewWords() total time 0.006421s
[DEBUG] (2018-07-16 11:42:56.395) VWDictionary.cpp:522::clear() 
[DEBUG] (2018-07-16 11:42:56.396) VWDictionary.cpp:522::clear() 25.804035               
[DEBUG] (2018-07-16 11:42:56.397) RegistrationVis.cpp:1303::computeTransformationImpl() 
[DEBUG] (2018-07-16 11:42:56.397) RegistrationVis.cpp:1316::computeTransformationImpl() words from3D=858 to2D=904
[DEBUG] (2018-07-16 11:42:56.398) util3d_motion_estimation.cpp:98::estimateMotion3DTo2D() words3A=858 words2B=904 matches=0 words3B=904 guess=xyz=0.000000,0.000000,0.000000 rpy=0.000000,-0.000000,0.000000
[ INFO] (2018-07-16 11:42:56.398) RegistrationVis.cpp:1346::computeTransformationImpl() Not enough inliers 0/10 (matches=0) between 1 and 0
[DEBUG] (2018-07-16 11:42:56.398) RegistrationVis.cpp:1664::computeTransformationImpl() transform=xyz=[null] rpy=[null]

[matlabbe]

Hi Terry,

When a guess is used, 3D features of the old frame are projected in the new frame (accordingly to guess), then correspondences are computed using a small radius around the projected position. When a guess is null, global matching is done between all descriptors. In your case, the global matching returned no match ("0 duplicated words added (from current image = 0)..."), thus transform cannot be computed.

How the first signature is created? Is it created using same parameters than RegistrationVis?

cheers,
Mathieu