In my experience there are three types of rider:
Much faster than you: Generally these guys fly past with such skill and speed that they don’t even need to call.
Much slower than you: Good chance these guys will pull over anyway, but otherwise I wait until there is ample room to pass because they can react very unpredictably when you call.
About the same speed as you: Simply no need to pass on singletrack.
Otherwise just use common sense. Don’t be pushy and rude, and try not to get too upset if someone else is. Remember, although it’s a race it’s not a matter of life and death, it’s supposed to be FUN!