Scaling does matter, though not as much as other aspects like whether you enjoy the subject and can do well.
Basically, higher scaling can compensate a bit for a slightly lower mark in a subject (unless it's something with phenomenally high scaling like maths ex 2, in which case you can get a pretty low raw mark but still band 6). In your case, chem has higher scaling than VA but if you're confident that you can do substantially better in VA then dropping chem would be a good decision. Another thing to note is that VA scales really badly if you get a low mark, but if you manage a mid to high band 6 then the scaling isn't as much of a problem (refer to the graph).
I'm not an expert, but usually, subjects that are taken by people who perform better overall have higher scaling. This could also indicate that the subject is harder, though.