Global evidence of men of all ages wearing baseball caps (some unto the grave) makes nonsense of your question.
Maybe you could ask:
‘How much should I care about what other people wear?’
or
‘When I dress, who do I want to impress?’
It certainly can’t be everyone, as opinions vary wildly. If you are really pushed for self-confidence, slap a cap on and then ask the one person in your life that you wish to impress most: ‘This hat, yes or no?’
And it must be worded exactly like that. Asking ‘do you like me in this hat?’ is far too ambiguous. Loaded, even.
And beware fashionistas who bitch about other people’s looks and recommend their own. They are the Nationalists of the clothing world.