I participated in a few – wouldn’t be right to say raced. Beards ago, mind.
I think back in the 90s you’d get big fields, with maybe the bottom half just out for a laugh. That’s steadily dwindled as MTBing has grown / fragmented, so nowadays an XC startline ranges from the fearsomely fit and strong, down to the reasonably fit and strong. There’s not much ballast like you’d see in an enduro, say.
You should definitely do it, though – it would be an awesome thing to be fit enough to mix it in an XC race, IMO. With a few CX races under your belt you sound well placed to give it a go.