Hi,
The main loop of rtabmap is the
Rtabmap::process(...) method. This can be a good place to start. It follows Algorithm 1 of the
paper (Appearance-Based Loop Closure Detection for Online Large-Scale and Long-Term Operation). Note that some stuff for RGB-D SLAM were added over the past years, but the memory management/loop closure steps are still the same.
cheers