As far as i'm aware, should he just turn up and expect to be let in, you have the legal right to tell him to come back when he has arranged a time.
It's a bit of a minefield, this area of things and, you can do what you like in the house really, as long as it's in the same condition when you move out, he can't moan.
If he does it again, tell him to bugger off and give notice, it's not on at all, and he knows it.
Is your deposit held by a third party BTW?