Loudenvielle is on a French bank holiday weekend next year and being the end of May should promise far better weather than the September races.
Fly into Toulouse, it’s 2 hours drive. The ski resorts on either side of the valley (Peyragudes at the top of the Gondola and Val Louron opposite) have loads of reasonably priced accommodation. If you’re moneybags or fast enough you’ll find somewhere at the bottom to stay.
Andorra from Toulouse is also doable, but the extra charges make it a bit of a mare. They may be low on tax, but they bill visitors for anything they can, Even O’Leary would be jealous of their enterprising spirit.