Don’t do it deep river!
I have huge sympathy with you. I’m a climber/mountaineer etc and get bloody pissed off by the same things. I started biking around 5 yrs ago now and actually ride much more than I climb as I have great riding on my doorstep (Surrey Hills) and no ‘proper’ crags within 3 hours (Swanage and Portland).
Now then, the trick is balance in these things. Just ’cause I ride more now than I climb, doesn’t mean I don’t actually love climbing a whole lot more. I gonna take a guess here that even though the riding gives you a buzz, it’s nowhere near the buzz you get from topping out on a great multi-pitch mountain route in the middle of nowhere? Cliche yes, but I feel most alive when I’ve been closest to death (dodgy gear, hanging belays, rockfall etc)
Re-think it – you’ll miss it mate!