Hi,
rtabmap requires the IMU quaternion (so yes only orientation) computed by
madgwick or
complementary filters (or imu directly if the imu provides already the quaternion). Here are some cases:
With gyroscope only, orientation will drift in roll, pitch and yaw over time. Can be still useful to track fast change in rotation for rtabmap's odometry nodes to guess next odometry transform (in particular for ICP odometry), but should not be used with rtabmap node, which uses it as gravity constraints in the map's graph (i.e., pitch and roll should not drift!).
With accelerometer only, orientation won't drift in roll and pitch, but in yaw over time. However, orientation changes may not be tracked accurately.
With accelerometer and gyroscope (6DoF IMU), the orientation won't drift in pitch and roll, but in yaw over time. The change of orientation will be more accurate than using accelerometer alone though.
This is the recommended minimal setup for rtabmap.
With accelerometer, gyroscope and magnetometer (9DoF IMU), then you have absolute orientation in roll, pitch, yaw (based on North). This setup is sensitive to magnetic fields though, in some cases (e.g., indoor) it adds more error in yaw though, so I prefer using 6DoF IMU indoor.
cheers,
Mathieu