Assuming that there are no restrictions that may be placed by your school on the maximum number of units that you can have, it would be good to take both Latin Extension and Physics. This would essentially allow you to make a more informed decision regarding the subject that you should keep because you will be able to compare your performance in each, instead of picking one over the other from the start and potentially ending up wondering whether you would have performed favourably in the other subject had you taken it. After dropping a subject, it will no longer count towards your HSC results or your ATAR, which can provide relief if your performance does not end up being at the standard that you were hoping (this applies more to Latin Extension).
On the other hand, if you cannot or do not want to take both Latin Extension and Physics, perhaps you should keep Physics. From your post, it seems that your performance potential in Physics is greater than in Latin Extension. At the same time, you can allocate additional study time towards Latin Continuers (allowing you to focus on addressing your areas of improvement), instead of increasing your workload by taking Latin Extension and reducing the study time available for Latin Continuers.
I hope this helps!