It is a great trip, and worth doing at least once. Like a number of the others, I've done it several times, although I haven't been for a few years now.
The drive down and back really is part of the fun; although we always went on A/B roads after Rouen to avoid funding the police Xmas party. We would head down early Thursday morning and then stop for lunch in a nice village somewhere before heading on to the circuit, setting up camp and heading to the local supermarket for supplies.
Friday morning is a great chance to drive a lot of the circuit, as they are switched back to public roads, and there are usually organised meets of some impressive exotica at specific locations. Friday evening in Le Mans town is the drivers parade which is good opportunity to see the drivers close-up, and to have one (or several beers). Le Mans has a lovely old town if you're feeling at all cultural.
Camping/facilities often leave a little to be desired, but it's funny how so many guys with expensive tastes in cars are prepared to put up with a few days in a dusty campsite in a rubbish tent with limited number of showers/toilets. It's all part of the overall character and madness of the event.
A grandstand ticket is not particularly expensive, and can be a good place to chill out away from direct sunshine if it's hot - the campsites don't usually have much in the way of tree cover.
Traffic around the circuit on Saturday lunchtime is horrendous - if you're going to go out then make sure you're back by mid-morning.
Taking an old bike is a nice way to see the far reaches of the circuit, particularly at night. Don't take anything special, a very cheap eBay option that just about rolls along would be perfect, and you could even leave it there afterwards if it's cheap enough.
The Pistonheads website always has a section devoted to Le Mans each year so you can pick up more useful info there.