I don’t think there’s any definitive answer. I do a lot of riding high up and exposed across the moors where 40mph winds are relatively normal. A straight headwind is a hard work, but manageable. It’s just like doing a long climb really. Tailwind is an absolute joy. But crosswinds, I hate them, and they begin to get pretty menacing at 40mph.
It’s not so much the winds, but the gusts. I certainly wouldn’t ride busy roads with gusts pushing me into the traffic. I’m lucky in most of the roads I ride are fairly desolate, but there are certain ones with powerful crosswinds where I spend as much time looking behind than I do forward, and I’m prepared to step off the bike and wait for traffic to pass if I see any!
Descending in cross winds is pure speed wobble mania in my own rigidly-fear-stricken experience. That’s no fun either.
But then you get back to low ground, and it’s fine… Until you pass a gate.
Overall, I think every ride needs to be judged on its own merits.
Once it gets up to around 70mph mind….no fun at all, and getting anywhere becomes near impossible. You spend more time with your feet on the ground trying to stay upright.