Geometry, tube diameters, butting etc… all will affect ride quality. Material is less important but does tend to drive the tube diameter, so much that they become intrinsically linked.
I love my alloy TCR – it is stiff
I love my steel fixed road bike – it is flexy
I love my carbon Defy – it is a stiff but smooths out vibrations
They are all set up to have the same position, have largely similar geometry, similar tyres (and wheels are swapped between alloy and carbon), and differ in weight by no more than 2.6 kilos. The Defy is the “best” bike as it is faster and leaves me less fatigued. The alloy TCR is a fine race bike. The steel bike is a nice ride. I too have had a Ti Merlin. The Defy beats it on every single ride characteristic. The TCR does too!