Just let them sit there, you da boss.
+1 aint that the truth baby!!!
But seriously, outside of a race/competitive ride it really ain't no thang unless you let it bother you.
Definitely don't do anything dangerous like braking hard or swerving, life's too short for that kind of lark.
Some people don't like the thought of an unknown rider drafting them for safety reasons. If thats the case, then just politely state that and ask them not to do it, we're all human beings capable of conversing with each other, right?