I would start to see if it could work hand-held first before going on the drone. Try this
example (RGB-D) or this
one (IR-depth) on the jetson nano. If VO rate is below 10 Hz, then it may be not possible to do, or you would need to find another VO approach (or try F2F optical flow odometry with parameters "--Odom/Strategy 1 --Vis/CorType 1 --Odom/KeyFrameThr 0.6") that is fast enough on Nano.