How do you compute the odometry? If you want to fuse visual odometry and IMU, look at the
robot_localization package, then set "yaw" parameter to false for the visual odometry config.
For 2D graph optimization, "yaw" is always corrected. You could revert all yaw values of the poses to original yaw values after graph optimization. Note that if your yaw is very precise, there should not be a lot of yaw correction anyway.
cheers