Are you taking Studies of Religion I or Studies of Religion II?
An informed decision in your case necessitates a balance between subject enjoyment/interest and favourable academic performance prospects.
Regarding History Extension, are you sure that the time you spent and the content you covered so far is sufficient in determining that History Extension would not be a subject that you want to take/keep in year 12? Have you considered giving this subject additional time and then determining whether or not to keep it? I personally believe that, if possible, giving this subject a bit more time will allow you to make a more informed and beneficial decision.
In terms of SOR, you mentioned it is a very safe subject. The emphasis here is on the word "safe" because it indicates a very positive assertion about this subject coming from you. The apparent problem is that you are unsure as to whether you will enjoy HSC SOR content as much as History Extension content, making this a more difficult decision to make. However, based on the way you described SOR (very safe) and considering the fact that you believe that History Extension will be highly demanding, I believe that academic performance prospects should be prioritised here. It definitely seems like you will be able to perform better in SOR than you would in History Extension, suggesting that you should drop History Extension.
It is also important to consider the fact that you are already taking Modern History and Ancient History, meaning that you will not miss out on a significant History component by dropping History Extension as you will still be taking 2 HSC History subjects. Lastly, I personally don't think you should drop Mathematics Extension 1 because considering the information you have provided (your performance in maths tends to fluctuate), you certainly have a good chance of ensuring consistently favourable performance in maths, which I believe would follow a much more clear, linear approach (i.e. study strategy/method) than that required for History Extension.
I hope this helps!