Transactional analysis: You initiated a conversation as a parent to a child (“Oi!”). And he responded as a child to a parent (“It wasn’t me so **** off”). You can break the cycle by communicating as an adult; you are more likely to receive an adult response in turn.
If I’d accidentally hit a neighbour’s car and they came out shouting “oi”, I wouldn’t stand there having a conversation with them. I’d apologise and say I’d be happy to speak to them when they’re calm.
It sounds as though you’re dealing with someone who’s emotionally a child. You got his back up with your aggressive tone. This has allowed him to have a grudge against you whilst taking no responsibility for his own part in the dispute. However hard it is, stay mature and polite.
So what did you do when this guy damaged your car then swore at you in the street? Because that’s a police matter. I’ve known people to be arrested for criminal damage because they bumped their neighbour’s bumper when parking.
If you don’t intend to do anything about him damaging your car and then swearing at you in front of the whole street, then I don’t see why you’d want to do something about him coming out of his house and looking at you.
My (honest and well-meant) advice would be to greet him politely every time you see him. “Morning!” and otherwise try to ignore him. Football chants? That’s not worth your effort to respond to, surely.
He wants to get to you so you need to show him that he hasn’t.