Not a fixed rule but typically, your fence is the one with the aris rail on your side.
Indeed. But uneducated nupties erect their replacement fences with the more anaesthetically pleasing side towards them.
Also, beware where ex. council properties are involved, all may not be as it seems. I bought an ex. Council house many moons ago and they sold me the neighbours brand new fence (she lived in a council rented property still) which her late husband had lovingly installed the wrong way round (aris rails to my side). So my property owned every fence, both sides and end (Councils like to sell off their fences…saves them having to maintain them).
Edit.
My boundaries were clearly marked on my deeds.