Stand in front your bike with your front wheel between your knees, then twist the bars from side to side.
Observe where the flex occurs.
Loose spokes, flexible stems, flexible bars and flexible forks can all contribute.
I noticed a big difference between when I trued the wheel, fitted a stiffer stem and fitted stiffer forks that also had a 20mm bolt-through.

