I switched from a normal school (I don't know where it's ranked, as most lists floating around don't include it) to a selective school (Merewether High, somewhere around #23) at the beginning of year twelve. If I had my time over I would have switched in year 11, because there were a few little things I missed out on (for example, I can't find anyone willing to vouch for my yr 11 extracurricular activities in my reference), but I've never regretted the switch itself.
Distance was a similar concern (I went from living in the same street as my school to a 45 minute bus trip), but I think it's outweighed by the benefits- selective schools can be more motivating and prevent complacency, because the competition is higher. The resources can be better. The exchange of ideas and discussions in class are generally more useful.
I would definitely recommend doing it. And if you start year 11 there and hate it, why not just switch back later?