6’4″ and I ride a large 19″ Alpine. Had a nerve am XL 22″ before that and a 20″ Cube Reaction before that .
After my Nerve I realised I really don’t want another XL frame. My Alpine fits me very nicely, im comfortable on it and after a few months with it I feel at home, can ride it well and it’s never caused me any discomfort in any area. That’s the real test IMO, does it cause you any pain or numbness or does it affect your abilities negatively in the mid to long term? If the answer is no then you’ve got a bike you can use.
Of course this is no use when buying a bike as you only get a short test ride, but I wouldn’t automatically assume you need an XL just because you are 6′ 4″ .