Maybe i could help at this point.
Today I've installed ROS and RTABMAP with g2o and gtsam.
I have carried out the following steps:
Install Boost:
sudo apt-get install libboost-all-dev
Source ROS:
source /opt/ros/kinetic/setup.bash
source ~/catkin_ws/devel/setup.bash
Install and remove rtabmap:
sudo apt-get install ros-kinetic-rtabmap ros-kinetic-rtabmap-ros
sudo apt-get remove ros-kinetic-rtabmap ros-kinetic-rtabmap-ros
Install g2o:
sudo apt update
sudo apt install libsuitesparse-dev
git clone -b c++03 https://github.com/felixendres/g2o.git
mkdir g2o/build && cd g2o/build
cmake .. -DG2O_BUILD_EXAMPLES=OFF
sudo make -j2 install
Install gtsam:
git clone https://bitbucket.org/gtborg/gtsam.git
cd gtsam
mkdir build && cd build
cmake ..
make check
sudo make install
Install RTABMAP from source (Dfreenect2_DIR is the path to my Kinect 2 libs, maybe you have to change or delete it):
cd ~
git clone https://github.com/introlab/rtabmap.git rtabmap
cd rtabmap/build
cmake .. -DCMAKE_INSTALL_PREFIX=~/catkin_ws/devel -Dfreenect2_DIR=$HOME/freenect2/lib/cmake/freenect2
make
make install
Install rtabmap_ros:
cd ~/catkin_ws
git clone https://github.com/introlab/rtabmap_ros.git src/rtabmap_ros
sudo chmod +777 devel -R
catkin_make
Thats my way to install rtabmap with g2o and gtsam. Maybe there are better or cleaner ways, I am also quite new to ROS. Path to the standalone Version of rtabmap is "/home/YOURUSERNAME/rtabmap" (just enter "rtabmap" in the terminal). rtabmap_ros is in the catkin_ws.
Best regards,
Wezza