Don’t forget, its not just “when riding”
When riding, most of winter I will go shorts, waterproof socks, base layer and jersey, warm gloves often loosing the base layer once warm.
If its windy and wet a shell will probably go on, but my “core” & legs rarely get cold.
What really will make the difference is, what happens when you stop, for eating, waiting for others or in the event of a mechanical/accident. Or are your rides a continuous non-stop affair.
No doom an gloom but, depending on where you ride, if your like me and only have what you “need” for riding and have to stop for whatever reason you may get very cold very quickly, if its an accident then this could quickly be trouble.
In my pack I carry a silver foil blanket for emergencies (me or someone else), and in winter will often have one of the small packable down jackets for when i/we stop for any length of time.
The number of times I’ve been glad of the down jacket, I’ve lost count.