I had the same problem. I think I knocked it with my knee once, so it had rotated 1/8 of a turn, and then it wobbled it’s way out and dropped off. Luckily the rider behind me spotted it and picked it up.
Now I use the silicon mount with an O ring looped through it. The O ring is then attached to some velcro that’s looped around my stem, it’s not noticeable once the garmin is fitted. I leave the silicon case permanently attached to the O ring and velcro and just put the Garmin in the case when I use the MTB (don’t bother with that security for the road bike)
It could come off the mount and come out of the silicon case at the same time but it’s pretty unlikely, given how well the silicon case holds onto it.