There is some math (mainly statistics) used in medical research which you can do as a research physician. But like @GrilledCheese said, very little in the clinical side of medicine, definitely no proofs and complex numbers.
You should do what you want to do, in this case, it seems like you want to do the double degree in mathematics and physics. You can always change after giving it a try. It is much better to do something and then regret it, rather than regret not doing it and wondering "what if".