I have this problem all the time when riding in the cold these days, has been affecting me for a few years now, didn’t use to. I am rather overweight, but also quite muscular so when working generate quite a bit of heat.
I don’t think it is down to blood sugar depletion, which is the most commonly espoused explanation. From what I can make out it comes down to the bodies systems attempts to regulate heat, and the affect of fat and age on the speed they can adapt to environmental changes.
Fat is considered as an insulator, and it kind of is, but it works both ways, it is perhaps better to think of it as thermal storage, when you work hard it stores heat, but when you get cold it stores the lower temperature. In itself it is inactive and cannot contribute to the bodies attempts to regulate temperature. So going from a warm enviroment after working hard it is in full on dump heat mode and doesn’t react quick enough to going into a cold environment, the core is still sending the message to dump heat when the skin (and fat)is getting cold
ps I am not calling you fat, but I seem to recall past posts where I thought you were a similar build to me. So suck it up fatty.
pps I am not a doctor.