I had one of them sort of problems the other night! Mine was an internal drive wheel nut though
Depends on how attached to the spacer you are - grind of the spacer (carefully) and then stilsons or simlar on the nut.
Or
Heat on the nut and a multi grip socket (if it will fit in the hole) if not carry on with the chisel
Good luck - horrid job!