I recently tried this. Went for a gravel bike and couldn’t get on with it offroad when the going got remotely tough, so now I’m on a Rigid 29er. (Not quite: I have a 27+ tyre up front at the moment, and still have a 29er wheel when I want to go quicker on smoother trails.)
I certainly was starting to think one bike to do it all is just a constant annoyance, but… I’m starting to think a do-it-all bike is a lot more feasible than I thought, you just have to go more towards the MTB end: an MTB with road wheels will ride a lot better on-road than a road bike with MTB wheels will ride off-road. Especially as you can’t even get the tyres in the latter.
So if I was going to trim down to one bike from my current two, I’d go for a rigid MTB and two sets of wheels, but I’d add two things:
1) I’d get a front MTB wheel that’s bigger than the back. Potentially a 29 or 27+ at the back, and 29+ at the front. Or perhaps better, a 27 at the back, and 27+ or big (but not ‘+’) 29 at the front. That way not only would I get away with rigid forks, but when I changed to road wheels the front end would drop down without having to adjust stem height or something, which would be annoying.
2) I’d stick with flat bars (not super wide) as drops off-road are too much of a compromise IMO. To get back multiple hand positions and long day comfort, I’d just have Ergon Gp2 grips or something instead.