I was going to speculate as to what you may have done to offend said delivery chap, but we know; nothing, as you have never met.
Is your Castle at then end of an annoyingly long drive? Do you have a velociraptor instead of a guard dog?
The fact that the undelivered parcel will have to be returned to you by the chap who is, even now, home early faking the delivery attempts, has a logical symmetry I find extremely satisfying.
I cannot wait to find out what his excuse is, I suspect that they (Hermes), tolerate a certain percentage of returns and he plays the system (as you do), but he could at least vary those addresses he does not deliver too.