i often consider this point. I actually enjoy what i do; what i don't enjoy is the stress that having to do it brings. By that I mean, I really need to keep this job in order to maintain the mortgage / std of living, etc. Hence when the other idiots ask me to do pointless and ridiculous things, I can't call them on it, for fear of losing the job. If I had the safety net of not really needing to work, I'd probably be more likely to say what I thought (which is probably right / justified anyway) which would make it even more enjoyable.
As a halfway (because being retired would quickly also become boring) I'd like to drive the DHL van. Listen to the radio, chat up receptionists all day, and still get a wage. Again, i know van drivers will tell me it's hard work and stressful too, but as above, not if you had enough money so you didn't actually have to do it.