As a mtb’er and ex motorbike green laner, I can see both sides of the story. However, what many ramblers,horse riders, mtb’ers etc. aren’t aware of, is they have somewhere around 100,000+ miles of of off road right-of-way, when users of motor based vehicles have less than 10,000. And what there is, is being downgraded year on year.
The damage they are capable of causing is/can be bad if not ridden/driven with restraint. However, when I used to go green laneing, most of the damage to land I rode on was caused by farm machinery of weather erosion.
And as for horses, they can do a fair bit themselves. Then when your finished riding one, you don’t just power wash it and leave it in the shed for a few weeks. You put it in field/paddock, where it chews that up and 5h1t5 everywhere.