I’d say wear it if you want.
If you wear it, one of two things will happen when you meet someone:
1) they won’t know what it is, so who cares.
2) they will know what it is, and if they recognise it they will almost certainly want to talk to you about it. At which point you can explain that your brother was in the regiment, he gave it to you and you’re wearing it out of honour and respect for your brother.
The logic I’m following here is, why would he have given it to you in the first place if wearing it would be a massive faux pas? If that were the case he’d surely have explained when he gave the gift, that it was a souvenir or symbol or whatever and you shouldn’t really wear it in public?