I find that sort of behaviour very very suspect.
There are few reasons which would justify entering a person’s home while they are out, and I doubt that any of those would apply here.
If you call the police and say “hi there, our neighbour’s boyfriend was found wandering around our house earlier today”, I reckon they’d treat it appropriately (i.e. have a word with the guy at least!).
On the other hand, depending on your relationship with your neighbour, you might want to chat it over with them first? The police arriving on their doorstep unannounced could be looked on unfavourably by them!