My rule for me….
Will it be a low speed mince – always casual (baggy)
Will there be sections of riding/pushing through undergrowth, snagging clothing etc – baggy outer layer, lycra snags to ****.
Will I stop and walk around with normal people (shopping etc) – baggy.
Will it be high speed but mainly gravity assisted – baggy
Else, lyrca rules – on or off road.
A quick riding trip to NZ & Oz and riding on the continent has led me to believe that it's only in the UK (and possibly North America- havn't been there but you get the impression from the mags) where we are hung up on xc/trail riding in baggy clothes being the only way to do it.