Gore (baggy) shorts are certainly not cheap, but imo they are not crap. They are very well designed and made, which makes them decent value for money in my books. I have many ‘hiking’ brand shorts, but I don’t think any of them would be much good as a long distance bike short (wrong cut, pockets in the wrong place, legs too short etc). I like Endura too, though they are a step down in terms of manufacturing quality. Many bike baggies are designed to be worn with a separate lycra inner/short underneath (which is exactly what you are suggesting anyway?!?). Both Gore and Endura sell baggies without inner shorts so you could wear your own choice anyway (I happen to favor Endura liners for the fit, which are also pretty cheap).
Also I have worn bike baggies for long hikes before as in some ways I like them better than many of my hiking shorts (TNF, Marmot, Mountain Hardware, Haglofs etc). I wore my Endura Humvess for most of the trip when I did the Inca trail a few years back.
You sure it’s not the price that’s clouding your judgement? 😉