I'm having similar thoughts. I love my Zed so much that I can't think of anything that I'd like to change it for, even though the cars I'm considering are probably better in different ways... 370, Cayman, M3, XKR, M6 etc.
I can't help thinking that dropping the extra cash into the 350 would make it even better. There's no guarantee that a different car would give you the same feeling. Although, I have heard some people worried so much about their cars after FI that it took the fun out of ownership... like every noise is your engine about to explode!
Have you driven an M3 yet? That might totally change your mind if not.