IF the car is legal and meets noise standards then you clearly have nothing to worry about.
If these people are genuinely threatening then perhaps some sort of action against them is required. Even just a letter explaining your point of view to them. Not a strongly worded threat of your own, just explaining your opinions, and that your car is perfectly legal.
If that doesn't stop them then council, police, ASBO, something like that would be the next step.
If it was you that was in the Kincardine shell at the same time as me on the way to KH for the trackday last month - I was in the 54 WRX that tailed you onto the M876 then into the petrol station - then I wouldn't have said your car was particularly loud.
Especially compared to some of the beasts on display that day.