As far as I know, frostbite has long term implications, but I’ve never heard of hypothermia doing the same.
With frostbite, you can kill the capillaries in your extremities. Afterwards, you have less blood-flow in those areas so they’ll be more likely to frostbite in the future.
A large part of how you deal with cold is psychological, so that might be a factor for you.
I’ve had frostbite myself and read a bunch of books about polar history and the medical effects of cold and never come across hypothermia as causing a long-term problem.