It sounds as though you’ve done the usual sensible things already.
I once spoke with a stalker who told me that there was very little chance of me disturbing a regular deer stalker as they tended to hunt at dawn, which is around stupid o’clock at this time of year. You could come across a bunch of posh kids on a shooting “party”, but they tend to be organised well in advance and are usually detailed on the “heading for the Scottish hills” web site so are easy to avoid.
Personally I think I’ve only been diverted by a shooting party once and considering it’s their money that pays for the trails I ride on year round, I’m happy to leave then too it. Much more likely to be diverted due to forestry operations though, in my experience.