I remember seeing a bug that Qt was crashing when trying to open a native Windows dialog. Which windows version are you using, and is it the latest rtabmap release? You may try
RTABMap-0.21.0-win64-old_deps version, which use different versions of the dependencies.