There is different “fitness” and I do know what you mean. Transferring from one type, 40 miles at a ploddy clip (roadies would consider 18mph average as a pottle) to short sharp sprints can show up your perceived unfitness.
I’ve never been good at endurance type activies. Short bursts of high output has always been my forte. I could play rugby at No 6 at a reasonable level for about 30 minutes. 50 minutes of pure hell and no use to anyone. Short hills, I can sprint up them at good times. Pass the bucket at the top. No good for anything for 5 minutes. Then I can do it over again. Recovery time is always pretty good.
To improve your “overall fitness” you have to train at different levels at different disciplines. For short burst fitness try squash or curcuit training. I used to train with the rugby club doing curcuits, squash, swimming and cycled everywhere. Was still no good at endurance events though.