I can see why you might want to change your route etc, but I really can't see how the driver can realistically "retaliate" without some kind of legal retribution.
You have his number, you have documented complaints to the company he represents.
I know many STW users have far more experience of this kind of thing than me, but I can't see it escalating.
I agree with Fullstop & Muddydwarf.