I feel the same, it’s very rare that I do 5 days of bike commuting cos I just end up knackered and not wanting to ride at the weekend.
Regardless of the comments about “take it easy one day”, it’s not always possible if you’re riding on busy streets and need your wits about you or you’re trying to keep up with traffic or if you need to be back for family commitments.
I do a round trip of 31 miles, sometimes on a mix of towpath/cycle track routes using the CX bike, when the weather is too bad to make that an option I use a SS road bike on a (fairly busy) road route. The first is more pleasant but takes longer, it can end up being 90 mins each way.
Just accept that you need a day off once in a while but make sure that you’re eating/drinking properly – if you’re stuffing yourself with junk calories it’ll make you feel far worse!