There are multiple ways to subscribe to image topic. When subscribe_rgb and subscribe_depth are true, you can do "rosnode info rtabmap" or rqt_graph to see which image topic it is subscribed to.
The features are extracted using one of the Features2D classes:
https://github.com/introlab/rtabmap/blob/master/corelib/include/rtabmap/core/Features2d.hMore precisely, when creating the signature:
https://github.com/introlab/rtabmap/blob/7be22d1b67c6a1f95e61cff0b66b9ef178d2c7ff/corelib/src/Memory.cpp#L4370-L4372