For the majority of riders I think age is less important and fitness is simply a result of time/effort put in. Unless you’re elite type level but guess most riders aren’t.
The fitest I’ve been was when I turned 16, racing track at national level. Then a serious motorbike accident stopped me riding properly for a year and never had the motivation to train that hard again so never been that fit again. My record at 15 was a 23.09 10TT, never been close to that since, not that I’ve tried either TBH.
I’m now 44 and been riding properly again for the last 10 years I’m starting to regain near the sort of form I was nearly 30 years ago! Main difference is that now with a family time on bike is limited and 30 years ago it was all I did, everyday!
Personally I have found the Chris Carmichael Time Crunched Cyclist approah to be very helpful and effective, focussing on power and not heart rate. For training 4 times a week and for at a minimum of 6 hours in total I can complete at a reasonable level. I’ll never win but given the hours input I’m very happy with the placements and results, and just my overall riding fitness having no problem banging out a 5 hours 100km XC race etc.