What you’ve done there is make the classic internet mistake of reading 4 words and ignoring the rest of what I actually said, It mostly depends on whether you’re doing one loop or laps.
If it’s basically XC, and you’re good, and you know what’s coming, having a rigid fork won’t slow you down much.
Where it makes a big difference for me is on unseen trails where that insurance clause of suspension can save you when it all goes wrong. If you’re cavalier, even that might not slow you down.
I don’t know everything, but I’ve ridden > 6ft drops and won races on rigid bikes, so I do base this on some experience.