Generally, physics is not a prerequisite subject for science related degrees (engineering maybe), if you are thinking of a science related degree, usually chemistry is the prerequisite (medicine, pharmaceutical etc.) Judging from your current selection of subjects, you seem to be be leaning towards a social-science degree so do whichever one you like better. However, if you are planning on doing a science degree, the skills learnt in chem and phys are super beneficial, so it really depends on yourself!
Heads up, VA is a lot of work (im not a reliable source tho, as i havent done VA myself, only heard it from friends who do), which has a major works (from what I've heard) and a lot of essays. If you do end up taking VA you will end up writing a lootttt of essays. Plus when it comes to these major work type subjects, continued effort is very important as you really can't cram it into one night.
On the other hand, i did do physics in yr 11 and i am continuing it for yr 12. I reckon physics in yr 11 gets a lot more conceptually interesting, and to be honest, the math required in physics is so easy that an ext 1 person will have no problem with it. The only problems you would encounter in physics as opposed to your other subjects would be the depth study, which if you do chem, your school might have them on at the same time which is a pain. However, since you have a tutor for physics this should be no problem. Another problem with physics is that usually school teachers tend to not teach physics very well, but this problem should be mitigated by your tutor.
Finally, for scaling. Physics and VA scaling is definitely very different. Although everyone says scaling doesn't matter, it really does... idk how the VA scaling works, but i can guarantee you would have to be in a higher percentile (e.g. 5%) as opposed to idk a lower percentile in chem and phys (e.g. 10%) to achieve the same score. However, I do know that chem and phys generally scale the same.
So, before you make your decision, i reckon you should be checking both of the textbooks out for both phys and VA and seeing which one you like better. (Perhaps try and do a quick summary of both of the contents for the year in the hols and maybe you will change your mind)
Quick question. If your school only allows 12 units for yr 11 how are you doing ext 1 math??? Wouldnt that make you have 13 units? Or do you have 11 units + ext 1 english or something???