https://www.askthe.police.uk/content/Q56.htm
Its a tricky one, as they need to obtain a court order to stop you in the long run.
Best way to think of it is, how would you act if you found someone cutting through your back garden, and then gave you a load of verbal when asked to stop.
Here's another blurb about it:
Trespass is a civil offence committed when somebody enters property where he/she has no right to be and refuses to leave when requested to do so by the owner, or his/her representative. In law, if a trespasser refuses to leave the property when asked, the owner/representative is entitled to use ‘reasonable force’ to evict him/her
‘Reasonable Force’ in Law
No attempt should be made to remove a trespasser unless he/she has first been asked to leave and has been given to do so. If he/she then refuses to leave voluntarily, ‘reasonable force’ may be used. What would count as ‘reasonable force’ is dependent upon the circumstances and unique to the situation but essentially requires deploying the absolute minimum force necessary. Any violence over and above what is absolutely necessary could leave the individual liable to prosecution for assault