Don't let the locals put you off Carpe diem. They're not all like that. The hill is used by walkers, horses, runners, XC and DH mountain bikers, as well as locals on scramblers quads etc. but from my experience, and I ride there regularly, there are very few problem interactions.
As a hill in the middle of a city, offering a variety of technical XC and DH lines I think it can't be beaten.
As Carzy Legs says though, we did hold a number of races up there and had a few problems with kids redirecting the course when the marshalls weren't looking. No muggings as far as I know though. Locals did steal a quad from the MTB Tour of Britain, and were tracked with the local police helicopter I think.
If you want some company for a ride, or to be shown the best trails, let me know.