If your not enjoying maths then chances are you won't want to do a career that involves maths... it's probably smarter to drop down now as there is less to catch up on. The peoples in my year who dropped down later took longer in to get used to the general course.
But if you think you need it...