hi sean, was the wheel straight(i mean align) when you hit the axle...?
lift the wheel a bit while you're hitting it(the axle..haha)...
i don't think you did anything wrong....
last time when i took mine out, i just gave the axle a few taps and the thing moved...
maybe you can try spraying some wd40 on the axle(or anything that's touching it)
hope this helps...