I ride in the NF alot since taking up road riding. It is on the whole 99.9% fine, I have had the occasional buzz near brokenhurst / beaulieu which though you can never prove it were obviously on purpose as there was plenty of space to pass safely.
I do tut to myself when a driver squeezes past too close and too fast and then is seemingly perfectly happy to slow down / stop and give a horse a massive berth.
Once in my 6000 miles or so have I been worried about hitting a horse when a herd of them were sprinting parallel along the roadside and then decided to turn and go straight across. I think I know who would have come off worse.
The forest gets ridiculously busy so I can see why residents get upset, however I'm sure caravans and tourists driving slowly is more of an issue than people on bikes...