speaking as somebody who spent 5 years of his life doing very little except for rowing at a fairly high level, I'd never class it as one of the hardest sports from a skill/technical perspective. Sure, the technique is critically important, but it's really not that difficult to learn. A lot of people who end up rowing at university or afterwards are those who were never sufficiently co-ordinated to be any good at any other sports: that should tell you something about the skill/natural ability level necessary. I loved it, though.
I reckon you need to include a "danger" coefficient as well. getting tired when you push yourself is one thing: risking serious injury or worse as you push the limits is quite another. so for me, no quasi-safe sport can possibly be the hardest.
therefore, my vote goes for the mountain stages in the TdF or back-country snowkiting (fly up, ride down). though DH racing isn't far behind. honourable mentions for alpine snowboarding (up to 100mph down a mountain with burning thighs and only one edge), boxing and football – everybody wants to play, so think how good you have to be to succeed – as well.