Kayaking is a brilliant sport. It has taken me to some of the most amazing places in the world and given me the most incredible experiences, both on and off the water. I've also met some really true friends for life through kayaking.
As others have said it compliments mountain biking very well, I often go away with boats and bikes and if the weather doesn't suit one sport it generally does the other.
The very, very best advice you will get is to get yourself down to your local club. They will be super pleased to help you get started and will have kit you can use until you decide what's best for you. The kit can be very discipline specific and you don't know which way you'll want to go until you've tried them (whether it's touring, white water, polo, sea, surf, freestyle, marathon, open, C1, tandem, the list goes on).
Please, please don't just go out, get some kit and try it unsupervised. The water can be a very dangerous place, and you really can't spot the hazards unless you know what they are.
UKRGB (www.ukriversguidebook.co.uk/forum) can be helpful, kind of in the same way people on here will be helpful if someone posted 'I'm new to mountain biking, give me some advice', so don't rely on everything you're told on there.
You could do a lot worse than getting yourself to Plas Y Brenin http://www.pyb.co.uk or Plas Menai http://www.plasmenai.co.uk and get on a weekend course, it really is money well spent getting yourself set off in the right direction. With canoeing/kayaking you really will get more out of the sport if you have the right basic skills, and you learn the right techniques from the start.
I'd also say it's pretty cheap too compared to Mountain biking. You could walk into a kayak shop and get all the latest, very top of the line kit, boat and paddles for less than two grand, you certainly couldn't do that in MTB. Not that you have to spend that much, it really can be pretty cheap to get into.
Most of all enjoy! You won't look back.