Fairly easy to do molgrips, as the distribution you'd need (XBMC) already has a couple of usable Pi ports.
But you'd need to couple it with (optionally) a case, a power supply (definitely), a display, a keyboard, possibly a WiFi module if you don't want to use the fixed ethernet port, and of course an amp and speakers.
That's a lot of kit to be sitting about, and while it would work, it would look uneasy on the eye, and would mean clicking with the mouse or keyboard every time you wanted to change a channel.
I have a Pi (had one of the UK's first) and do a lot of hacking with it, and what I'd do instead is get myself a Pure One Flow- £80 or so quid. Sounds great, is great, all in one compact unit.