I think it's important to consider that some of the negativity you might be receiving while driving your Z might be completely unrelated to the fact that you're driving a Z. However, if you have the pre-conceived notion that people might be negative towards your Z, you'll attribute any negativity you get, to your Z.
I'd say looking back over my last few years of ownership I've had much more positive than negative attention. It's certainly an attention grabbing car. Children either don't notice it or absolutely love it - I've had kids embarrassing their parents by running over and asking me to rev it - that happens quite often. Teenage chavs are a mixed bag, and probably the only ones I've ever received any negative attention from, though nothing significant at all. I just put it down to them either being jealous or just trying to show off in front of their mates. I had one chav shout over "NAH I DONT LIKE YOUR CAR MATE", which I took as a complement to be honest. I've found dads and older adults will stare in interest quite a bit. I've had older adults talk to me about it a few times, usually asking what it is and being shocked that it's such a relatively old car, always very positive.
Of course, if you're driving like a dick then everyone's going to think you're a dick. But that goes for any car.