If you’re out on a long off road ride, you usually have a bag, so can stuff your jacket in there if things get a bit too warm.
On a long ‘serious bizniz’ road ride (you know, the ones where the rider is convinced they’ll be picked for the Tour someday), no pack will be worn, so if the conditions change the rider will have no option but to carry on, sweating in the heat and unable to rehydrate because his single, quarter full bottle has run out.
I’ve started dumping the bag for most road rides myself these days (don’t tell anyone, ok?), and always wear a base layer with huuge pockets so I can stuff a waterproof or windproof in.
Less stops on the road, so less variation in temperature, but roadie clothing is always a compromise – I tend to err on the side of wearing too much, as I prefer to be slightly too warm than too cold.
Or I just slow down and take the tourer, put the panniers on, take a spare layer and stop for a picnic half way round. Much more civilised.