He sounds mental. Our neighbours have a right of access up the side of our house and across our backyard. We stuck a gate on the drive and use a combination lock. We put their bins out for them. Our car gets moved if they need access for bigger things. We’re happy, they’re happy and their toddler can’t sneakily get out and into the road.
It may not be fair, but have you considered clubbing together and offering to pay for a second gate? It might not be fair or right, but it would be a lot less hassle. Stick combination locks on both and agree that people won’t cross his garden frivolously. The thing with the fence panel just sounds insane. Offering to pay for the gate might even shame him into doing it himself, though probably not.
ps: My advice would be to move to Glossop, which ime is altogether more civilised and less full of massive idiots :-/