I almost always wear mine… They’re no bother to wear, they don’t restrict me or overheat me or chafe me (unless I forgot to clean them!), there’s just no downside at all to it for me.
(Knees don’t do an awful lot of heat radiating- that’s why you can wear shorts or a kilt in weather where you wouldn’t want short sleeves)
I think people fixate on the serious injuries, but I’ve had enough little scrapes that have just caused inconvenience, or kept me off the bike for a couple of weeks, or just been a bit sore and spoiled a ride…
So I ask myself, why not, and don’t find any why nots.