Splines on the crank arm will be worn.
You may be able to save it by cleaning it all up and tightening the clamp bolts to the top of the acceptable torque setting* – it’s a lot tighter than you think if doing it by hand. The bolts need doing up alternately until they are *both* at the right torque.
If they aren’t done up tightly enough in the first place, the crank arm moves slightly and becomes looser as it wears.
The preload bolt is just that – it’s to ensure that the slack is taken out of the setup before the crank arm clamp is tightened. It doesn’t perform any function in terms of keeping the arm on.
(*Probably 12 – 14 Nm, but check the Shimano manual for your cranks.)