I’m not keen on most road style gear, and the fabric they are made of, so only have a couple of jerseys with those funny little pockets on the back. If it’s a lycra-fest (road ride or 24hr race etc), then I’ll drag them out, although when on a MTB I tend to tape or attach tools and pump etc to the bike, rather than risk them popping out if I get bounced off a drop etc.
Much of the MTB riding I do involves hike-a-bike sections, or lifting bikes over deer fences. I don’t like having tops that are only useful for riding, so most of my upper body gear is used for walking, running, skiing, biking, etc. So, I tend to just wear a thin wicking baselayer, possibly with a lightweight hard or soft shell if its wet/cold/windy. Roadie style tops are pretty rubbish here, so I habitually carry a Wingnut bag, Camelbak or similar. Means I can carry a few survival bits and pieces, spare lights, food, water bottle etc (ideally not on the frame – except if it’s a lycra-fest, I don’t like muddy nipples)