I had a similar problem, which eventually resulted in the crank coming off mid-ride. Problem was that the splines on the axle were worn, so no matter how tight the pinch bolts were, the crank would slip off – only fix was a new axle (which really gave me an excuse to get a new chainset 😉 )
The pre-load cap isn’t there to keep the crank on, it’s purely to get the crank pushed on far enough in the first place – it should only be done up finger-tight, and then maybe even backed off a bit. It’s the pinch bolts that hold it on. I broke a couple of the preload plastic caps as the crank tried to come off whilst it was screwed in.
Have a look at the splines on the axle and the inside of the crank – if they’re shiny, like the finish has come away, that’s probably your issue