Suprisingly no-one has replied to you yet lel, I was going to but I thought it would be better for someone who legit knew what they were talking about lol
I googled a bit, and I think I sort of know the answer
Here are some references I found useful
https://www.physicsforums.com/threads/displacement-and-work.322461/
http://physics.stackexchange.com/questions/184659/work-force-x-distance-vs-displacement
https://en.wikipedia.org/wiki/Conservative_force
basically, I think that there is work done on the object because you pushing it is not a conservative force, so at each infitesmal portion of the path, you have changed the kinetic energy. I understand that work can be negative only with a conservative force so when you throw a ball up against gravity and catch it, the net work done in that system is zero, however with a non-conservative force like friction, it merely adds up at each portion of the journey (I think)
Take all this with 500 tons of salt though, not entirely sure that I am correct