I’d say only a small number of segments have an unbelievably difficult KOM to beat (buy someone quicker than me of course)
There is a bit of setting times by not taking ‘the right line’ or riding down a trail adjacent to the segment or using a motorised device etc. Although I didn’t think people did that last one until cannondaleleking admitted to it!
If enough people ride the trail enough times, sooner or later a fast person is going to get a GPS error in their favour at the start and end of the trail. Which makes things difficult for us mere mortals!
Also perhaps your opinion of what’s fast is limited by the people you know. Even if you think of the fastest person you know locally, there is almost certainly someone faster in the area. When you throw in 5 plus years of Strava and people travel around. I’m sure someone ‘mind blowingly quick’ has also ridden that trail.
At least these are the excuses I keep telling myself 😉