This is not complex.
If you want to tell them to get off (I would) then tell them, I think you’re within your rights.
If you aren’t bothered then fine!
Neither camp has to justify to the other camp what is more acceptable.
Folks are laid back about different things.