Definitely not a waste of time.
I've dropped off the gym in the last 5 months, and have noticed my upper body aches considerably more following a longish (6hr+) ride now compared to when I was doing a weights routine 3 or 4 times a week. Granted, I've become faster on my bike due to the increased time I've gained by losing the gym, but for a 24 hr, I reckon it's essential. It's just a compromise that's needed!