Hi,
RTAB-Map is mostly an integration of multiple SLAM modules or existing algorithms that have their own mathematical derivation. In particular for graph optimization theory, look for TORO, g2o or GTSAM papers. For loop closure detection, look at this paper:
M. Labbé and F. Michaud, “
Appearance-Based Loop Closure Detection for Online Large-Scale and Long-Term Operation,” in IEEE Transactions on Robotics, vol. 29, no. 3, pp. 734-745, 2013. (IEEE Xplore)
For motion estimation with visual features, look for Perspective-N-Points (PnP) approach and bundle adjustment theory. For motion estimation with lidar, look for Iterative-Closest-Points (ICP) papers, in particular libpointmatcher.
cheers,
Mathieu