How stiff your frame is, eg thickness of chain stays and what size tyres/pressure you were running?
Tyres are important, yes. Frame stiffness also, but….
See also;
Bars and stem – Something with some flex or damping can really help here. Good carbon, or even ti, bars are great at soaking up a bit.
Seatpost – See above
Saddle – See above
Tape – As others have said, good tape is the shizzle.
Frame layup – Not just stiffness, but the way the frame is designed to work. Yes, we all know that laterally stiff and vertically compliant is the marketing guff to end all marketing guff, but actually (hold on!) it’s what you want in a frame for your needs here. Good handling, with good comfort.
Wheels – You’ve sort of answered this yourself already. A set of aero-grrrr race wheels will be fast, but they won’t be comfy. A set of nice hand built audax-spec wheels will be fast AND comfy.
Many factors to consider here, and that’s before we even look at fit! For example, a layback seatpost. Maybe, but it might upset your positional balance overall and then you’re screwed.
As others have added, if you’re not racing, why ride a race bike? You wouldn’t, I assume, take a full on hardcore XC race bike out on your local chilled trail loop. So, why would you take a MOAR AERO GRRRRR race road bike out for a ride?
If you’re not racing….