Depends how bad you feel.
I ride every day (commute) and don’t stop for anything really, especially chesty ones. If I’m feeling ropey I just rein it in a bit.
I find getting breathless helps clear out the crap and usually feel better for it.
Before I commuted I’d have a chesty cold once a year that last a fortnight or so, and I’d take it easy to accommodate it.
Not had a bad one in 5 years and I don’t think its coincidence.
Swollen glands, etc are a different though I’ve heard. Sign of proper impending grogginess so maybe best take it easy when that happens.