there's been some actual science done on this.
Objective: To determine whether the author’s 20.9 lb (9.5 kg) carbon frame bicycle reduced commuting time compared with his 29.75 lb (13.5 kg) steel frame bicycle.
Design: Randomised trial.
Setting: Sheffield and Chesterfield, United Kingdom, between mid-January 2010 and mid-July 2010.
Participants: One consultant in anaesthesia and intensive care.
Main outcome: measure Total time to complete the 27 mile (43.5 kilometre) journey from Sheffield to Chesterfield Royal Hospital and back.
Results: The total distance travelled on the steel frame bicycle during the study period was 809 miles (1302 km) and on the carbon frame bicycle was 711 miles (1144 km). The difference in the mean journey time between the steel and carbon bicycles was 00:00:32 (hr:min:sec; 95% CI –00:03:34 to 00:02:30; P=0.72).
Conclusions: A lighter bicycle did not lead to a detectable difference in commuting time. Cyclists may find it more cost effective to reduce their own weight rather than to purchase a lighter bicycle.
if i read it right: the biggest factor was weather. Windy andor rainy days lead to slower times.