Look at your technique, keep your heels down and you shouldn’t really come off
All well and good but you could liken this to the smash technique also!
Riding SPD’s off road is harder than flats, however the benefits are massive. Think of it as having an extra set of hands, you can ride lighter, position the bike, pedal more efficiently etc etc.
It will take longer to master than flats and yes you will have some offs on the way, however once mastered you wont even think about unclipping, it will happen as fast as getting your foot off a flat pedal, but it also shouldnt need to come off as much either.
The one thing id miss is the floating technique, where you can lift your heels and float over things when it wets rough, its hard to describe but you can unweight the bike rather than smash through things meaning you are far less likely to have an off or get a puncture and its faster.
Each to their own in the end but its easier to give up than persevere, however the benefits are massive