if you have used a standard star-fangled thingy, and it is newish (so not corroded) it is possible that the threaded centre part of it is not crimped fully, and is spinning in the rest of it. The centre part sometimes loosens during installation if the thing is driven home using a bolt threaded into the centre, i.e. so that the load is not applied to the springy bits.
Suggestion; if you have access to an air impact wrench, or similar, try using that, whilst applying pressure from below (eg by holding the stem (with loose bolts) whilst bearing down on the bike).
If all else fails I would suggest cutting away one of the spacers with a hacksaw; this may then allow you to free the parts by some devious means.
cheers