I(f(x)) = integral of f(x) dx
I(x^2*tan^3(x))
= I(x^2*tan(x)*sec^2(x)) - I(x*tan(x))
= x^2*tan^2(x)/2 - I(x*tan^2(x)) - I(x*tan(x))
but I(x*tan^2(x))
= I(x*sec^2(x)) - I(x)
= x*tan(x) - I(tan(x)) - x^2/2
= x*tan(x) + ln(cos(x)) - x^2/2
As for I(x*tan(x)) - it boils down to integrating ln(cos(x)), which I can't see any way to do using HSC methods[1]. I'd put it in the box of questions too hard to solve in the HSC.
[1] or first year uni methods for that matter. There might be a couple of second-year techniques that could be applied, but you obviously aren't going to be expected to know them.