Hi,
To use g2o optimization, rtabmap should be built with it, otherwise the option in Preferences is disabled.
With rtabmap.launch, you can select g2o by doing this:
roslaunch rtabmap_ros rtabmap.launch rtabmap_args:="--Optimizer/Strategy 1"
To find the corresponding parameter in ROS from the Preferences panel, you can mouse over the parameter box (spinbox, checkbox...) and a tooltip should appear with "Optimizer/Strategy" for example if you mouse over the combobox to select graph optimization strategy. Another way is to show parameters in terminal:
$ rosrun rtabmap_ros rtabmap --params | grep g2o
Param: Optimizer/Robust = "false" [Robust graph optimization using Vertigo (only work for g2o and GTSAM optimization strategies). Not compatible with "RGBD/OptimizeMaxError" if enabled.]
Param: Optimizer/Strategy = "2" [Graph optimization strategy: 0=TORO, 1=g2o and 2=GTSAM.]
Param: g2o/Optimizer = "0" [0=Levenberg 1=GaussNewton]
Param: g2o/PixelVariance = "1" [Pixel variance used for SBA.]
Param: g2o/Solver = "0" [0=csparse 1=pcg 2=cholmod]
cheers