Assuming the car weights were the same, and the power levels were the same, then yes the M3 would be quicker. It's the chassis that will do it, and in terms of feel the E46 is one of the best and most neutral cars I've ever driven (and given how bloody good the Z33 is, that's high praise!). It's likely the driver in front of you was crap, but then you also don't know how hard he was pushing or whether the car was 100% or level of experience etc. I refer back to the video I posted some years back of me in an MR2 all over the arse of a 360 at Anglesey: Now there's no way that's down to the car, it's got to be down to the driver. Trackdays are a terrible way of comparing cars unless you're willing to swap keys with someone else to do back to back runs.
And that S54 is one of the all-time great engines. The DE (or even HR) cannot get anywhere close to it, it's like comparing a filet mignon with a Big Mac.
Keep everything the same including drivers, track, day, weather etc and the E46 M3 will be quicker. More enjoyable? Perhaps, certainly I think so but there's an awfully good case for the 350Z to be equally or more so. I personally wouldn't buy an E46 over a 350Z, but I won't argue it's the better car.