I've found my average speed is really variable, depending on the weather (windy / wet), riding solo or in a group, and especially how much climbing is involved (usually lots round here!)
It's been more useful to join a club and see where I 'fit' relative to a large number of people. The fastest guys and girls in the club kick my arse in terms of average speed (but then they are very, very good!), I'm 'middle of the pack' riding with the 'intermediate' group of guys, and I'm quicker than the slower group.
On average, gender and age start to have an impact when you look at the fastest racers, but for most of us it's possible to become a "good intermediate" rider with sufficient effort.
But I think it's easy to become obsessed with the 'numbers', as in the end I just ride because I love it