I wouldn't buy team kit purely because I could not wear it in races. I assume that is the reason it is looked down on. Basically you should nbe wearing your own club kit.
I wouldn't wear a leaders or champions jersey because in my mind that is a bit like walking around with a replica Olympic medal around your neck.
Still, plenty of people buy team kit, some of it looks really nice and at the end of the year it can be had cheaply. I would probably mock a mate but if you just turned out to ride with me I would think nothing of it.
Maybe avoid the full kit and just stick with black shorts?