It depends, some events you get to pick your start time, so if you are near the start you don’t have to wait around much.
PMBA series was set up like this last year, and the slower riders generally set off first.
Some are a free for all, so if you don’t get to the start of stage 1 early, you’ll be waiting round for people to set off. I did hamsterley this year, and was near the back and had to wait around 30mins at each stage. The track also got chewed up more so you were at a bit of a disadvantage, was my excuse anyway :)