The Party Wall Act only covers what is in the title, walls on a boundary that may or may not be part of a structure and any excavations that affect them (principally undercutting foundations but can also be used if backfilled)
Boundary disputes are a civil matter, but any damage could also be seen as criminal damage.
In either case you should have been notified of any works and asked permission to proceed beforehand.
If the party wall is in effect, common practice is for each party to have a surveyor, both paid for by the person undertaking the work. If an agreement can not be reached, a third party wall surveyor is employed to mediate.
A neighbour should be entitled to access via your property, if you refuse access they would need to make a legal case, otherwise they are trespassing.
CAB may be your best first port of call.
CAB link
Take a multitude of photos of the damage, with reference points in the background.
Save the most recent photos of the garden before the damage was done.
Good luck.