I bought a pair of Endura Zyme shorts for this exact use, but damn it, they’re actually too good to use on the bike and I ended up wearing them as casual wear for most of the summer!
Being honest, I wear my road lycra most of the time on the gravel bike as it’s practical. On the days when I need extra pockets (or need to be seen in public!), I chuck a pair of Gore C3 Trail over the top of the bibs – I’ve worn Gore baggies for years and really rate them. Obviously, being Gore, they are overpriced but if you can find some for a decent discount, they are nice shorts to wear.