I think there is a secret to it.
Its best to do it early morning when there's not too many walkers about. I normally seem to have more control than they do (also they've trampled less wet mud onto the rock)
Take it slow, and aim for the middle of the trail – the rocks are bigger, but they're actually fixed into the ground so you've only got the tyres slipping on the rock to deal with, not the rocks moving at the same time. That all said, don't try and be too picky about line choice – its more a question of avoiding the worst bits, than picking the easy bits. Once you've got going, just keep it rolling. Trying to get started again, get clipped in etc is a nightmare.
I normally get my worst scares on the grass at the bottom – you feel you can go fast, there's some nice jumps. Then you realise there's even less grip than the rocky bit…
It's certainly a good challenge!