When I used to work outside all day, I’d be fine most days but there would be days where the cold would feel like it had got into your bones and took an age to get warmed up when you got home.
No amount of clothing seemed to make a difference. Have had similar long days on the bike in the winter.
But generally I think ‘feeling’ cold is a mental toughness thing. You can train yourself to be less affected by it (the cold shower thing is a good example of it).