I'm afraid I've been there, done that and although it's satisfying, violence is NEVER the answer. You're clever enough to know that the right thing to have done would have been to look him in the eyes, say nothing and walk away (and then come and tell the story on STW). It is always hard not to react though, especially when someone has quite literally threatened your life.
It was my wife who was on the receiving end in my case, she'd been out for a run, got barged, spat at and called various names, came home in tears, I got on my bike and tore after them in a rage – ended up decking 3 of them. I think I've told the story on here before. Yes it was very satisfying. No, it wasn't the right thing to do. The police were called. I ended up on a charge, nearly lost my job over it.
I'd still 'tweak' my route home just in case he (or the police) are waiting to 'follow it up'.