The only other reason not to wear proper cycling gear is vanity – the 'I don't look good in lycra' syndrome, but if that's your attitude then you might as well just wear a sign saying 'i'm not happy with my shape'. Why not? Nobody else is offended.
Unfortunately, some others are a little off put by it; I work with mostly women (and 2 of the 4 other blokes are gay, so who knows what to think!! 😯 😆 ) and some of the girls do get a bit embarrased, so I tend to wear baggies when I commute.
Baggies are a bit more 'practical' if the ride ends at the pub, cafe, pizza place afterwards as we don't have trail centres near us with 'on-site' facilities, so you're mixing with the 'general public'. Again, I don't care, but others do; besides I don't mind baggies or lycra so no problem for me.
The chamois in most baggies are pretty good these days too, where as previously you had to get roadie stuff to get a decent pad.
TBH as long as you are comfy, go for it.