Shorts/trouser pocket for me.
It sits pretty much on top of my thigh rather than to the side if that makes sense, so not really in danger from a crash. You can see many of the semi-pro guys (i.e. the ones who wont have a mechanic and physio meeting them at the finish line) at a downhill world cup doing the same and they crash constantly.
If it makes any difference, I prefer my riding shorts and trousers to be quite tight fitting (not the “baggies” of the early 2000’s) and it goes on the side of my trailing leg, purely because thats the side I always have my phone.