If it is the same person visiting (bubble) or the occasional very minor transgression then let it go. Anything more and I’d say that you have a responsibility to stop it as they are endangering others, stalling societal/economic recovery, etc.
Whether you do this by police or going round is up to you. No it isn’t how I would like society to be but neither are your neighbours behaving in a responsible manner.
Anybody who suggests “live and let live” or similar has obviously failed to grasp the seriousness of the situation or hasn’t been directly affected. From personal experience, close friends in the NHS are on their knees in tears due to overload and neighbours parents have died. More broadly in my area children are missing education, further damaging life chances, pre-scool kids aren’t getting socialised so may have problems later in life, older folk or vulnerable are suffering from mental health issues due to loneliness.
Stick to the rules, it is the only way that we are going to get out of this mess any time soon.