The parameter is not set at the right place. If you modify rtabmap.launch, set it under rtabmap node
here:
<param name="Grid/RangeMin" value="5"/>
The more general way I use with rtabmap.launch is to do:
roslaunch rtabmap_ros rtabmap.launch args:="-d --Grid/RangeMin 5"
When you change a parameter with "rosparam set" after the node has been launched, you should call "update_parameters" service of rtabmap node so that rtabmap knows it needs to re-read the parameters.
cheers,
Mathieu