I always let a faster rider pass at the earliest opportunity and expect others to do the same for me if I catch them up.
I can't keep riding knowing I'm holding someone up. Its wrong.
Also, why spoil someones fun going downhill when you're on a climb and could probably do with a short rest?
General rule I follow and hope others do too, always give way to the faster bike, up or down.

