I seriously think it's road biking – 6 odd hours of pedalling at fairly close to threshold. tactics, downhilling @ 70mph, mental toughness.
That said, at my level of road biking it's none of these things 🙂
another 'tough' sport is solo yacht racing (vendee globe etc). you've got to be a tough, tough bastard to be able to cope, both physically (cold, infection, getting battered by various bits of the boat) and definately metally (isolation, knowing if anything goes wrong, there are some locations where you know you'll never be found, self belief, meteorology, tactics, sailing skill)
out of sports making my 'f*ck that' list Solo RTW yachting is my number 1.