I agree that it would be a bit rude to take a photo of you walking at 5 yards without asking, but as others have said, it's totally impractical for him to ask permission, and no harm's done to you (really).
He's taking a photo because you're doing something interesting and which is a challenge to capture well, not because you are somebody interesting.
Vaguely related story which might give another perspective: My dad's garden back onto a big field where people walk their dogs. I was taking photos with a wide angle lens and waited for a guy with a dog to get to an aesthetic position within the field and took a few photos. You can barely see him in the photo, just that he's there in what's quite an abstract photo. He rang the doorbell 5 minutes later, really quite angry, demanding I tell him why I was taking his picture. Because he was so arsey, my inital reaction was really indignant: "Get over yourself! I don't know who you are, and I don't care! I'm not taking pictures of you, I'm taking pictures of a guy walking a dog in a field." Didn't say it quite like that, though.
Here's a question, nuke: Assume it's a technically good photo – well composed, well timed, in focus; Would you be happier if, when he's sorting the day's pics, he puts your photo in the "good" pile, or the "meh, boring" pile?
i reckon there's a little bit of photo slut in all of us.