Of course it matters. Reasonable cause can be twisted to allow access to anything, and as the terrorist fear takes over it just gets worse. The law has now made it a legal requirement for ISPs to keep your browsing history for two years: What's stopping them doing the same with your driving data? And then making it a requirement for insurers to inform the police of any speeding or spirited driving? As the accident rates go up, it's an easy push for the hand-wringers of the world to say they're saving the children by taking speeding drivers off the road.
If you can't afford a fast car without a black box, then get a cheaper slower car until you're a bit older. That's what people have been doing for years. I don't think the push towards boxes in insurance is inevitable, as if people don't use them then insurers won't offer them.
Sorry man, I know this is coming across as a bash at you, but I really don't mean it that way. It's more a rant against the system in general, as I see it as a very real threat to fun motoring.