Bag of Binary Words VS RTABMap

[leo152]

Hi!
   I want to use loop closure detection method, and tried Bag of Binary Words as well as RTABMAP, however, I found that BoBW only needs 10ms to process a frame, while RTABMAP typically needs 1000ms or more, RTABMAP is also using ORB feature, the same as BoBW.
   Since both of these two methods are using bag of words method, why RTABMAP is much slower than BOBW? Even though the recall curve of RTABMAP is better than BOBW.
   Is there any one can introduce the difference between the two methods?

Best Regards

[matlabbe]

Hi,

not sure how you got the timings for RTAB-Map, but if I do a simple test with the images used for demo_surf in DLoopDetector project, I got DBoW2 42 ms/image and RTAB-Map 35 ms/image (using both SURF64). Note however that on larger datasets, RTAB-Map would have an average time higher than DBOW because it uses an incremental vocabulary. An incremental vocabulary requires an extra step to update its structure as new visual words are added. This would be the biggest difference in timings between the two approaches. An incremental vocabulary and a fixed vocabulary have both advantages and disadvantages.

Here is what I did:

$ git clone https://github.com/dorian3d/DLoopDetector.git
$ cd DLoopDetector
$ mkdir build
$ cd build
$ cmake -DBUILD_DemoBRIEF=ON -DBUILD_DemoSURF=ON ..

DBoW2:

$ ./demo_surf 
DLoopDetector Demo
Dorian Galvez-Lopez
http://doriangalvez.com

Loading SURF64 vocabulary...
Processing sequence...
Adding image 0: ./resources/images/SVS_L_1235603737.305831.png... 
init done
- No loop: All the images in the database are very recent

Adding image 1: ./resources/images/SVS_L_1235603738.305924.png... 
- No loop: All the images in the database are very recent

Adding image 2: ./resources/images/SVS_L_1235603739.306133.png... 
- No loop: All the images in the database are very recent

Adding image 3: ./resources/images/SVS_L_1235603740.306344.png... 
- No loop: All the images in the database are very recent

Adding image 4: ./resources/images/SVS_L_1235603741.306427.png... 
- No loop: All the images in the database are very recent

Adding image 5: ./resources/images/SVS_L_1235603742.306720.png... 
- No loop: All the images in the database are very recent

Adding image 6: ./resources/images/SVS_L_1235603743.306852.png... 
- No loop: All the images in the database are very recent

Adding image 7: ./resources/images/SVS_L_1235603744.306946.png... 
- No loop: All the images in the database are very recent

Adding image 8: ./resources/images/SVS_L_1235603745.307141.png... 
- No loop: All the images in the database are very recent

Adding image 9: ./resources/images/SVS_L_1235603746.307355.png... 
- No loop: All the images in the database are very recent

Adding image 10: ./resources/images/SVS_L_1235603747.307446.png... 
- No loop: All the images in the database are very recent

Adding image 11: ./resources/images/SVS_L_1235603748.307648.png... 
- No loop: All the images in the database are very recent

Adding image 12: ./resources/images/SVS_L_1235603749.307869.png... 
- No loop: All the images in the database are very recent

Adding image 13: ./resources/images/SVS_L_1235603750.308080.png... 
- No loop: All the images in the database are very recent

Adding image 14: ./resources/images/SVS_L_1235603751.308135.png... 
- No loop: All the images in the database are very recent

Adding image 15: ./resources/images/SVS_L_1235603752.308376.png... 
- No loop: All the images in the database are very recent

Adding image 16: ./resources/images/SVS_L_1235603753.308589.png... 
- No loop: All the images in the database are very recent

Adding image 17: ./resources/images/SVS_L_1235603754.308666.png... 
- No loop: All the images in the database are very recent

Adding image 18: ./resources/images/SVS_L_1235603755.308885.png... 
- No loop: All the images in the database are very recent

Adding image 19: ./resources/images/SVS_L_1235603756.309106.png... 
- No loop: All the images in the database are very recent

Adding image 20: ./resources/images/SVS_L_1235603757.309189.png... 
- No loop: All the images in the database are very recent

Adding image 21: ./resources/images/SVS_L_1235603758.309390.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 22: ./resources/images/SVS_L_1235603759.309609.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 23: ./resources/images/SVS_L_1235603760.309697.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 24: ./resources/images/SVS_L_1235603761.309911.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 25: ./resources/images/SVS_L_1235603762.310118.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 26: ./resources/images/SVS_L_1235603763.310198.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 27: ./resources/images/SVS_L_1235603764.310405.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 28: ./resources/images/SVS_L_1235603765.310624.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 29: ./resources/images/SVS_L_1235603766.310829.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 30: ./resources/images/SVS_L_1235603767.310899.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 31: ./resources/images/SVS_L_1235603768.311120.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 32: ./resources/images/SVS_L_1235603769.311337.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 33: ./resources/images/SVS_L_1235603770.311422.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 34: ./resources/images/SVS_L_1235603771.311645.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 35: ./resources/images/SVS_L_1235603772.311858.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 36: ./resources/images/SVS_L_1235603773.311942.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 37: ./resources/images/SVS_L_1235603774.312146.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 38: ./resources/images/SVS_L_1235603775.312361.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 39: ./resources/images/SVS_L_1235603776.312452.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 40: ./resources/images/SVS_L_1235603777.312724.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 41: ./resources/images/SVS_L_1235603778.312878.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 42: ./resources/images/SVS_L_1235603779.312970.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 43: ./resources/images/SVS_L_1235603780.313174.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 44: ./resources/images/SVS_L_1235603781.313381.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 45: ./resources/images/SVS_L_1235603782.313599.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 46: ./resources/images/SVS_L_1235603783.313738.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 47: ./resources/images/SVS_L_1235603784.313889.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 48: ./resources/images/SVS_L_1235603785.314103.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 49: ./resources/images/SVS_L_1235603786.314192.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 50: ./resources/images/SVS_L_1235603787.314396.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 51: ./resources/images/SVS_L_1235603788.314623.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 52: ./resources/images/SVS_L_1235603789.314708.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 53: ./resources/images/SVS_L_1235603790.314914.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 54: ./resources/images/SVS_L_1235603791.315129.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 55: ./resources/images/SVS_L_1235603792.315219.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 56: ./resources/images/SVS_L_1235603793.315570.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 57: ./resources/images/SVS_L_1235603794.315630.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 58: ./resources/images/SVS_L_1235603795.315718.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 59: ./resources/images/SVS_L_1235603796.315938.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 60: ./resources/images/SVS_L_1235603797.316140.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 61: ./resources/images/SVS_L_1235603798.316361.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 62: ./resources/images/SVS_L_1235603799.316437.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 63: ./resources/images/SVS_L_1235603800.316655.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 64: ./resources/images/SVS_L_1235603801.316861.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 65: ./resources/images/SVS_L_1235603802.316961.png... 
- No loop: There are no matches against the database (few features in the image?)

Adding image 66: ./resources/images/SVS_L_1235603803.317164.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 67: ./resources/images/SVS_L_1235603804.317369.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 68: ./resources/images/SVS_L_1235603805.317472.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 69: ./resources/images/SVS_L_1235603806.317687.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 70: ./resources/images/SVS_L_1235603807.317879.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 71: ./resources/images/SVS_L_1235603808.317972.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 72: ./resources/images/SVS_L_1235603809.318188.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 73: ./resources/images/SVS_L_1235603810.318390.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 74: ./resources/images/SVS_L_1235603811.318481.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 75: ./resources/images/SVS_L_1235603812.318701.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 76: ./resources/images/SVS_L_1235603813.318902.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 77: ./resources/images/SVS_L_1235603814.319117.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 78: ./resources/images/SVS_L_1235603815.319201.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 79: ./resources/images/SVS_L_1235603816.319409.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 80: ./resources/images/SVS_L_1235603817.319623.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 81: ./resources/images/SVS_L_1235603818.319715.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 82: ./resources/images/SVS_L_1235603819.319925.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 83: ./resources/images/SVS_L_1235603820.320149.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 84: ./resources/images/SVS_L_1235603821.320219.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 85: ./resources/images/SVS_L_1235603822.320438.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 86: ./resources/images/SVS_L_1235603823.320648.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 87: ./resources/images/SVS_L_1235603824.320733.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 88: ./resources/images/SVS_L_1235603825.320943.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 89: ./resources/images/SVS_L_1235603826.321150.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 90: ./resources/images/SVS_L_1235603827.321243.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 91: ./resources/images/SVS_L_1235603828.321453.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 92: ./resources/images/SVS_L_1235603829.321674.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 93: ./resources/images/SVS_L_1235603830.321875.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 94: ./resources/images/SVS_L_1235603831.321962.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 95: ./resources/images/SVS_L_1235603832.322186.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 96: ./resources/images/SVS_L_1235603833.322391.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 97: ./resources/images/SVS_L_1235603834.322471.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 98: ./resources/images/SVS_L_1235603835.322692.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 99: ./resources/images/SVS_L_1235603836.322902.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 100: ./resources/images/SVS_L_1235603837.322975.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 101: ./resources/images/SVS_L_1235603838.323210.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 102: ./resources/images/SVS_L_1235603839.323398.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 103: ./resources/images/SVS_L_1235603840.323492.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 104: ./resources/images/SVS_L_1235603841.323705.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 105: ./resources/images/SVS_L_1235603842.323925.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 106: ./resources/images/SVS_L_1235603843.324006.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 107: ./resources/images/SVS_L_1235603844.324212.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 108: ./resources/images/SVS_L_1235603845.324433.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 109: ./resources/images/SVS_L_1235603846.324632.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 110: ./resources/images/SVS_L_1235603847.324724.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 111: ./resources/images/SVS_L_1235603848.324957.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 112: ./resources/images/SVS_L_1235603849.325145.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 113: ./resources/images/SVS_L_1235603850.325235.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 114: ./resources/images/SVS_L_1235603851.325447.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 115: ./resources/images/SVS_L_1235603852.325673.png... 
- No loop: No match reaches the score threshold (alpha: 0.3)

Adding image 116: ./resources/images/SVS_L_1235603853.325753.png... 
- No loop: No temporal consistency (k: 1). Best candidate: 6

Adding image 117: ./resources/images/SVS_L_1235603854.325965.png... 
- Loop found with image 6!

Adding image 118: ./resources/images/SVS_L_1235603855.326169.png... 
- Loop found with image 8!

Adding image 119: ./resources/images/SVS_L_1235603856.326246.png... 
- Loop found with image 11!

Adding image 120: ./resources/images/SVS_L_1235603857.326471.png... 
- Loop found with image 14!

Adding image 121: ./resources/images/SVS_L_1235603858.326685.png... 
- Loop found with image 14!

Adding image 122: ./resources/images/SVS_L_1235603859.326756.png... 
- Loop found with image 16!

6 loops found in this image sequence!

Execution time:
 - Feature computation: 39.2444 ms/image
 - Loop detection: 3.09127 ms/image

RTAB-Map (with parameters from Benchmark wiki page, without "-Kp/IncrementalFlann false" but with "-DbSqlite3/InMemory true -Mem/BinDataKept false" to avoid hard-drive and image compression overhead):

$ rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Kp/MaxFeatures 400 -Kp/BadSignRatio 0.25 -Rtabmap/TimeThr 700 -Kp/DetectorStrategy 0 -DbSqlite3/InMemory true -Mem/BinDataKept false resources/images 

Deleted database "rtabmapconsole.db".
rtabmap init time = 0.087043s

Parameters : 
 Data set : resources/images
 Time threshold = 700.00 ms
 Image rate = 0.00 s (inf Hz)
 Repeating data set = false
 Camera starts at image 0 (default 1)
 INFO: All other parameters are set to defaults
   Overwritten parameters :
    Bayes/FullPredictionUpdate=true
    DbSqlite3/InMemory=true
    Kp/BadSignRatio=0.25
    Kp/DetectorStrategy=0
    Kp/MaxFeatures=400
    Kp/TfIdfLikelihoodUsed=false
    Mem/BinDataKept=false
    Mem/RehearsalIdUpdatedToNewOne=true
    Mem/STMSize=30
    RGBD/Enabled=false
    Rtabmap/PublishStats=false
    Rtabmap/StatisticLogged=true
    Rtabmap/StatisticLoggedHeaders=false
    Rtabmap/TimeThr=700
    Rtabmap/WorkingDirectory=.

Processing images...
 iteration(1) time=0.034429s/0.040767s
 iteration(2) time=0.036473s/0.042607s
 iteration(3) time=0.036903s/0.042962s
 iteration(4) time=0.036776s/0.042939s
 iteration(5) time=0.037535s/0.043495s
 iteration(6) time=0.038713s/0.044671s
 iteration(7) time=0.037774s/0.043983s
 iteration(8) time=0.040416s/0.046509s
 iteration(9) time=0.042959s/0.049009s
 iteration(10) time=0.039085s/0.045172s
 iteration(11) time=0.040853s/0.046842s
 iteration(12) time=0.041029s/0.047510s
 iteration(13) time=0.039403s/0.045573s
 iteration(14) time=0.042311s/0.048499s
 iteration(15) time=0.041133s/0.047189s
 iteration(16) time=0.040704s/0.046874s
 iteration(17) time=0.036573s/0.042562s
 iteration(18) time=0.038013s/0.043993s
 iteration(19) time=0.037248s/0.043298s
 iteration(20) time=0.037032s/0.043324s
 iteration(21) time=0.037158s/0.043318s
 iteration(22) time=0.038625s/0.044681s
 iteration(23) time=0.038177s/0.044433s
 iteration(24) time=0.040527s/0.046894s
 iteration(25) time=0.042014s/0.048307s
 iteration(26) time=0.042459s/0.048756s
 iteration(27) time=0.041656s/0.047844s
 iteration(28) time=0.040043s/0.046053s
 iteration(29) time=0.044147s/0.050153s
 iteration(30) time=0.036045s/0.041924s
 iteration(31) time=0.038882s/0.044727s
 iteration(32) time=0.033945s/0.039831s
 iteration(33) time=0.034621s/0.040422s
 iteration(34) time=0.032961s/0.038921s
 iteration(35) time=0.033332s/0.039181s
 iteration(36) time=0.030589s/0.036416s
 iteration(37) high(1) hyp(0.05) time=0.028684s/0.034424s
 iteration(38) high(1) hyp(0.08) time=0.029477s/0.035873s
 iteration(39) high(1) hyp(0.09) time=0.024527s/0.030314s
 iteration(40) high(1) hyp(0.09) time=0.025643s/0.031210s
 iteration(41) high(2) hyp(0.10) time=0.024698s/0.030170s
 iteration(42) high(2) hyp(0.10) time=0.024885s/0.030311s
 iteration(43) high(2) hyp(0.10) time=0.022707s/0.028182s
 iteration(44) time=0.021630s/0.027275s
 iteration(45) time=0.020828s/0.026164s
 iteration(46) time=0.021785s/0.027207s
 iteration(47) time=0.020345s/0.025698s
 iteration(48) time=0.019744s/0.025131s
 iteration(49) time=0.018255s/0.024137s
 iteration(50) time=0.016570s/0.022157s
 iteration(51) time=0.015985s/0.021336s
 iteration(52) time=0.014813s/0.020440s
 iteration(53) time=0.016783s/0.022088s
 iteration(54) time=0.015142s/0.020458s
 iteration(55) time=0.015070s/0.020410s
 iteration(56) time=0.014813s/0.020282s
 iteration(57) time=0.014685s/0.020200s
 iteration(58) time=0.014644s/0.021972s
 iteration(59) time=0.015031s/0.020289s
 iteration(60) time=0.014311s/0.019880s
 iteration(61) time=0.014146s/0.019535s
 iteration(62) time=0.014990s/0.020449s
 iteration(63) time=0.014740s/0.020132s
 iteration(64) time=0.015159s/0.020659s
 iteration(65) time=0.017863s/0.023442s
 iteration(66) time=0.014969s/0.020597s
 iteration(67) time=0.016802s/0.022445s
 iteration(68) high(3) hyp(0.07) time=0.022321s/0.028072s
 iteration(69) high(3) hyp(0.06) time=0.029576s/0.035370s
 iteration(70) high(31) hyp(0.05) time=0.035088s/0.040870s
 iteration(71) high(31) hyp(0.05) time=0.035115s/0.041035s
 iteration(72) high(31) hyp(0.05) time=0.035857s/0.041638s
 iteration(73) high(31) hyp(0.05) time=0.033723s/0.039499s
 iteration(74) high(31) hyp(0.05) time=0.032711s/0.038728s
 iteration(75) high(31) hyp(0.05) time=0.030897s/0.036840s
 iteration(76) high(31) hyp(0.06) time=0.032828s/0.038716s
 iteration(77) high(10) hyp(0.06) time=0.032117s/0.037939s
 iteration(78) high(10) hyp(0.06) time=0.034064s/0.040006s
 iteration(79) high(9) hyp(0.05) time=0.033714s/0.039483s
 iteration(80) high(13) hyp(0.05) time=0.034270s/0.040128s
 iteration(81) high(26) hyp(0.06) time=0.032865s/0.038694s
 iteration(82) high(26) hyp(0.07) time=0.035096s/0.041020s
 iteration(83) high(23) hyp(0.08) time=0.037884s/0.043733s
 iteration(84) high(23) hyp(0.08) time=0.039954s/0.045840s
 iteration(85) high(27) hyp(0.07) time=0.038993s/0.044974s
 iteration(86) high(27) hyp(0.06) time=0.040513s/0.046547s
 iteration(87) high(26) hyp(0.06) time=0.042246s/0.048193s
 iteration(88) high(26) hyp(0.05) time=0.041371s/0.047382s
 iteration(89) high(29) hyp(0.06) time=0.047496s/0.056704s
 iteration(90) high(29) hyp(0.06) time=0.049369s/0.055795s
 iteration(91) high(29) hyp(0.07) time=0.044865s/0.051047s
 iteration(92) high(29) hyp(0.07) time=0.046168s/0.052379s
 iteration(93) high(26) hyp(0.08) time=0.045468s/0.051599s
 iteration(94) high(27) hyp(0.07) time=0.045179s/0.051669s
 iteration(95) high(27) hyp(0.07) time=0.043788s/0.049997s
 iteration(96) high(23) hyp(0.07) time=0.046014s/0.052097s
 iteration(97) high(26) hyp(0.09) time=0.050375s/0.059391s
 iteration(98) high(26) hyp(0.10) time=0.050510s/0.056783s
 iteration(99) high(27) hyp(0.09) time=0.049879s/0.056081s
 count = 100, loop closures = 0, max time (at 98) = 0.050510s
 WM(51)=[1,0;2,0;3,0;4,0;5,0;6,0;8,1;9,0;10,0;11,0;12,0;13,0;14,0;15,0;16,0;17,0;21,3;22,0;23,0;26,2;27,0;28,0;29,0;30,0;31,0;32,0;33,0;34,0;35,0;36,0;38,1;39,0;40,0;41,0;42,0;43,0;44,0;45,0;46,0;47,0;48,0;49,0;50,0;51,0;52,0;53,0;54,0;55,0;56,0;57,0;58,0]
 iteration(100) high(27) hyp(0.07) time=0.047779s/0.054191s
 iteration(101) high(27) hyp(0.06) time=0.047176s/0.053551s
 iteration(102) high(29) hyp(0.06) time=0.046074s/0.052227s
 iteration(103) high(12) hyp(0.06) time=0.043659s/0.049968s
 iteration(104) high(12) hyp(0.06) time=0.043419s/0.049656s
 iteration(105) high(23) hyp(0.06) time=0.039558s/0.045668s
 iteration(106) high(23) hyp(0.06) time=0.042123s/0.048163s
 iteration(107) high(14) hyp(0.06) time=0.042651s/0.048659s
 iteration(108) high(21) hyp(0.05) time=0.043605s/0.049755s
 iteration(109) high(23) hyp(0.07) time=0.044311s/0.050304s
 iteration(110) high(21) hyp(0.07) time=0.046718s/0.052783s
 iteration(111) high(21) hyp(0.07) time=0.047965s/0.054044s
 iteration(112) high(17) hyp(0.07) time=0.049063s/0.055083s
 iteration(113) high(13) hyp(0.06) time=0.051197s/0.057300s
 iteration(114) high(4) hyp(0.07) time=0.053587s/0.059971s
 iteration(115) high(14) hyp(0.07) time=0.051314s/0.060210s
 iteration(116) high(15) hyp(0.08) time=0.046062s/0.052490s
 iteration(117) high(8) hyp(0.10) time=0.044617s/0.050828s
 iteration(118) loop(8) hyp(0.12) time=0.044810s/0.050978s *
 iteration(119) loop(9) hyp(0.13) time=0.047269s/0.053330s *
 iteration(120) loop(12) hyp(0.14) time=0.046801s/0.053018s *
 iteration(121) loop(13) hyp(0.16) time=0.048430s/0.054469s *
 iteration(122) loop(15) hyp(0.19) time=0.044056s/0.050304s *
 iteration(123) loop(17) hyp(0.25) time=0.037873s/0.037921s *
Processing images completed. Loop closures found = 6
 Total time = 5.055005s
 Cleanup time = 2.295741s
Database ("rtabmapconsole.db") and log files saved to current directory.

Using showlogs.m (see Benchmark page) to compute the average processing time with the output files LogF.txt and LogI.txt, this gives (with MatLab or Octave):

$ showlogs;

Loading log files...

meanTime =  34.792
maxTime =  0.053211
maxDict =  8056
maxWM =  71
LTM =  100
meanWM =  26.512
meanDict =  4071.1
meanWordsPerSign =  284.14

cheers,
Mathieu

[matlabbe]

I did a longer example with CityCentre dataset (FAB-MAP dataset). Here DBoW2 finds a lot of wrong loop closures before image 680 (around where the second traversal begins), example:

...
Adding image 492: ./resources/images/0493.jpg... 
- Loop found with image 151!

Adding image 493: ./resources/images/0494.jpg... 
- Loop found with image 133!
...

Execution time:
 - Feature computation: 97.8672 ms/image
 - Loop detection: 18.0362 ms/image

With RTAB-Map using the same command line as in the previous post, results are very similar to the paper in terms of Precision/Recall (no wrong loop closures before image 680), but timings are different (faster computer and "Kp/IncrementalFlann=true"):

$ showlogs;

Loading log files...

meanTime =  103.82
maxTime =  1.2797
maxDict =  305987
maxWM =  1206
LTM =  1237
meanWM =  590.07
meanDict =    1.5500e+05
meanWordsPerSign =  399.42

Note the max time (1.3 sec). It is where the vocabulary structure needs to re-updated (kd-tree balancing). We see better on this plot (the 5 peaks show where the structure is updated, only when the vocabulary doubles in size):

[leo152]

Hi!
    Thanks for your response.
    I found that in the Benchmark example, the official output of rtabmap-console of CityCentre, it cost 1300s to process 1237 images. Is that because the two options "-DbSqlite3/InMemory true -Mem/BinDataKept false"?
    When processing the CityCentre dataset, timeLikelihoodCalculation cost 100ms per image and timePosteriorCalculation cost 50ms per image. As a result, it may cost 150ms per image to calculate loop closure detection?
    In the original paper of BoBW, it claimes that Bag of words processing time is below 20ms when processing 26292 images as shown below. It may cost 30ms to deal with features, but only 20ms per image to process loop closure detection in a large dataset(26292 images).

[matlabbe]

Hi,

The biggest difference between the timings above and those in the original RTAB-Map's paper is the parameter "Kp/IncrementalFlann" (which didn't exist at the time the paper was made). For example, if I set "-Kp/IncrementalFlann false" like in the Benchmark page (to reproduce exactly the experiment), we got the ~1.4 sec processing time as you mentioned. The vocabulary is re-constructed after each image with "Kp/IncrementalFlann=false". Note that in this case the acquisition time was set to 2 seconds and the time limit was set to 1.4 sec. The limit could be set to 700 ms for example, so that maximum processing time would be bounded around 700 ms, independently of the number of images processed.


Beside the inherent costly vocabulary structure update of RTAB-Map, if we look only at the time of the likelihood/posterior computation (the bag of words part on the DBoW figure you've shown), we get ~40 ms for likelihood and ~70 ms for posterior (total ~110 ms, 85% Recall at 100% precision). There are some new options to reduce this time though. If I set "-Kp/TfIdfLikelihoodUsed true" (TF-IDF search approach) and "-Bayes/FullPredictionUpdate false", they can be reduced to ~0.8 ms and ~8 ms respectively (total ~9 ms) after 1237 images, while keeping high Precision/Recall (86% Recall at 100% precision in this case). If we interpolate linearly to 25000 images and if we ignore memory management that could bound those times, we would have ~16 ms likelihood and ~176 ms posterior computation time, which is indeed higher than the ~10 ms of DBoW. Here is a breakdown (note that I re-used "Kp/IncrementalFlann=true" here to save some time repeating the same experiment):


cheers,
Mathieu

[leo152]

Hi!
   I really appreciate for your response. There is still some problems I cannot figure out.
   
  1. For incrementalFlann, if you use this method, the vocabulary should be reconstructed after each image. However, it appears that if  Kp/IncrementalFlann is set false, the vocabulary is updated after each image.
  2. We get the same results with / without  Kp/IncrementalFlann, so why do we need this time-consuming module?
  3. I tried the parameters you metioned, and the input is like this :
./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0  -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/TfIdfLikelihoodUsed true -Bayes/FullPredictionUpdate false -Rtabmap/TimeThr 1400 -Kp/DetectorStrategy 2 -Kp/NNStrategy 3 -Kp/NndrRatio 0.9 ~/Documents/data/Images_CityCentre/_joined/
    I used orb features since it's fast to compute, However, the computation time reached 1s per image at the 400th iteration. Other NNStrategy performs poorly for binary features.
    When I changed to surf features, the runtime is pretty steady at around 300ms per image.

[leo152]

Hi,
   I also noticed that posterior also take a lot of time. However, this bayesian inference cannot give perfect solution for multiple loop closures. For example, if loop closure happens at 1st, 100th, 200th, 300th frame, their probability will be lowered.
   I mean that such bayesian inference do helps to improve the recall of public datasets and remove outliers, but are they really helpful in real world applications?

Best Regards
Leo

[matlabbe]

Hi,

1. With "Kp/IncrementalFlann" false or true, the vocabulary is still incremental, new words are added to vocabulary after each image processed. This parameter indicates if we want to reconstruct the vocabulary (rebalancing) only after it doubles in size (=true) or after every time we add new words (=false). There is a parameter called Kp/IncrementalDictionary (along with Kp/DictionaryPath) that if set to true, an offline pre-built vocabulary is used (avoiding the structure update of the incremental vocabulary), but it is not very tested and the vocabulary creation is not documented.

2. Kp/IncrementalFlann set to true or false may not change Precision/Recall results, only processing time. As I said, this was an optimization done after writing the paper. However, when "Kp/IncrementalFlann=true", it is more difficult for memory management to correctly estimate the total processing time (as if a structure update happens, it will take a lot more time than the previous updates) so that it can correctly make sure that processing time will never get over the acquisition time (keeping loop closure detection online).

3. RTAB-Map lacks of a dedicated efficient structure for binary descriptors. I cannot tell you what are the best parameters for binary descriptor as I didn't test/benchmark them in large experiments. You can still use KDTree Kp/NNStrategy=1 with Kp/IncrementalFlann=true for ORB descriptors, that would reduce computation time, but not sure about Precision/Recall results. For this reason for loop closure detection with RTAB-Map, I would stick with SURF.

cheers,
Mathieu

[matlabbe]

Hi,

Do you mean if the same loop closure happens on 1st, 100th, 200th, 300th frame (same location)? The Bayes filter will sum up probabilities together for many images of the same location linked by loop closures. In practice, I didn't do the test using the likelihood output directly instead of the Bayes filter, so it is difficult the answer this question.

cheers,
Mathieu

[leo152]

Thanks ! Really grateful for your response. I got the idea of "Kp/IncrementalFlann" now.
I tested ORB features and with Kp/NNStrategy=1 and got a recall of 40% approximately.

Best Regards
Lei Han

[leo152]

Sorry for not making this question clearly.
I mean for example, the 1000th image is at the same location of 1st, 100th,200th,300th image. Suppose the likelyhood between the 1000th image and kth image is L_k.
L_1 = L_100 = L_200 =L_300. Then for the 1000th image, p(S_t = 1) = p(S_t = 100) = p(S_t = 200) = p(S_t = 300) will be very small because of the normalization process. Will it be a problem?
Such a situation do happen when a car is driving along a circle for many laps.
Best Regards
Lei Han