Hi,
I didn't based the odometry approaches implemented in rtabmap on a particular paper, I took the basic ideas from Frame-To-(Key)Frame or Frame-To-Map approaches of many visual odometry papers and implemented them with available functions included already in OpenCV, PCL and g2o (e.g., feature extraction, feature matching, ransac registration, 3D->3D or 3D->2D estimation, local bundle adjustment...). See this
visual odometry survey paper (Scaramuzza and Fraundorfer) for overview.
cheers,
Mathieu