It is not extremely difficult to understand concepts if you have decent teacher and read the textbook. But the most troll part of physics is possibly the fact that they give you curveballs questions(situations where you haven't met before) and expect you to draw upon knowledge learnt in the course