In STW stylee I will quote the theory I have learned recently rather than give you the benefit of any true experience (or success)
Anaerobic fitness is more quickly gained and lost – 3-4 weeks
Aerobic fitnees takes time to build up but is therefore lost more slowly – 3 months
It’s possible to peak too early/overtrain
Really good XC racers or people doing short max efforts have a really good aerobic base
Taking the above into account some way of building your base fitness before heading into speed phases is a sound idea (you can build the speed in less time and closer to race season but you can’t do that with the base). How you do it is open to your own interpretation of the myriad things you will hear. In my case I’m making use of my 16-20mile commute to get quality (power meter) Z2 miles in. If you do enough and intersperse with something more unstructured and longer at the weekends you will find they become quite tiring.
Take care your training doesn’t become more important than race performance – I’m in danger of this!