I think track cycling came about because at the time (late 19th century) the roads were terrible and you couldn’t ride very fast on them, compared to a smooth wooden track.
Plus it’s a much more enjoyable spectator sport on a track of course, money aside. Watching a cycle race from the roadside without TV or radio is rubbish.
As for it being silly – no sillier than athletics or swimming, is it? You’re racing the other people. And you can end up shoulder to shoulder with your opponent, unlike MTB racing usually. It’s all about the competition, and if you’re not competitive then forget it.
If you are though, it’s the cycling equivalent of speedballs.