The only downside is they look ridiculous.
You’re riding a bicycle. To most other people (if you are in the UK,US or Aus) you are a grown-up on a child’s toy or a ridiculous mamil (or both)
As for other cyclists/MTBers/peers, I don’t *think* that they look at your pogies in the freezing cold and then loudly deduct points from your imaginary fashion score 😂 Well, they might, but then again they are wearing turquoise and orange onesies in an attempt to look like a Teletubby who is Considerubbly Richer Than Yow. It all gets weird at this point.
OP, pogies are only ‘overkill’ if it’s too warm. Especially on the hill-climbs Which in my experience is most of the winter. But I am SO glad of them in driving freezing rain or sleet, or in very occasional biting freezing winds. I do sometimes stow them in backpack or panniers and re-fit at the top of hills for the longer descents. YMMV? I’m in the English Mids and I tend to run on the warm side.