Just to muddy the waters, I’m 5’6 with a 54cm diverge (on the same sizing guide as the Tarmac, Roubaix, Venge, Allez and Langster).
The guide puts me on a 52cm. The shop had a 49, a 52 and a 56cm in. Their advice was that the 52 was a little small (it felt very small), and the 56 was too big (felt OK coming from a MTB only background).
They ordered in a 54cm for me, and while it initially felt a little small, I now think it is spot-on. Very comfortable. Apart from saddle height, I haven’t felt the need to change anything on it.
I would caution looking at the actual geometry, though, as reach, etc. doesn’t seem to scale linearly with frame size (for Specialized, at least).