I didn't test ride my first proper mountain bike, apart from a couple of laps of the car park. I thought it was fine and it was, until I started getting into more technical riding.
At the time though, even if I had taken it off road for a day, I would have found it perfectly fine for the type of stuff I was riding.
Buying a new one this time around though, I definitely wanted to test ride the bikes and as my choice was fairly mainstream it wasn't too hard to find demo bikes. I couldn't get hold of a Zesty to demo so discounted it.
In the end, I test rode a 2010 Stumpjumper & a 2010 Fuel EX9. Did a first lap on the Stumpjumper and the second lap on the EX9. Within about 5 mins of getting on the EX9, I was 100% convinced that it was not as good for me as the Stumpjumper was.
So, for me a test ride was definitely worth it.
I can see how it might be difficult though, if you are hoping to test one frame over another, as the overall build will probably change the feel of the finished bike quite a lot.