Go and ride a big MTB. Then go and ride a BMX. The BMX will no question feel more 'chuckable' – i.e. changes direction quicker, easier to throw around, easier to hop/jump. OK this may be somewhat to do with geometry, but part of it definitely the physical size of the fame.
Yes – the difference between a MTB and a BMX are rather large – hence the choice of tool for job. However, the difference between a L and XL in a frame are so minimal on the 'chuckability' front that it can't be more than a couple of hundred grammes difference. Example – Turner 5 Spot geometry:
2010-Turner-5-spot
Those numbers are not that far apart, the contact points are going to be similmar distances apart, the wheel base will be similar. So, between a S and an XL there might be a decent difference, but there is almost nothing between L and XL.
I get 'chuckability', when it is the difference between a BMX and a MTB. I don't get it when people claim the differences up or down a frame size. The numbers aren't different enough.