"He’s just a little boy, don’t demonise him for having rubbish parents"
this. He clearly prefers to be at yours not his , this in my experience is cyclical and depends on interests and wider circle. When I was a kid my parents safe garden and unsafe shed were a magnet to all my friends in summer, someone else's massive but tatty house with easily accessed roof voids and frankly negligent parent's for winter . Many of my friends spent all day every day at my house as teenagers including one who would turn up sit in the front room and try and bond with my dad by smoking a pipe of Tabaco.
Also friendship as children can be weird, an overheard row does not mean your son is being bullied they could be best mates within minutes.
Having said that If your son does not want the kid round back him up and be firm.