"was given the allotment a far few years ago after a particularly nasty incident"
So what!
And in any case he was given his allotment, not yours.
Here's 2 realistic options. If he keeps trying to take your patch then surely he is breaching the rules? In that case escelate it with the council and make it very clear to them that you will take it right to the top. It will be him that gets kicked off the allotments.
Other option, tell him you would be happy for him to give you a hand in assisting you with your lot, but remind him it's yours.
In fact I would just go with the first option. Maybe take it to the point where he gets a stern warning.
If he keeps doing it then take it all the way – he only has himself to blame.
And get down there more, do a bit of research and find out what are the easiest plants to keep etc…