I use a pair of Kyle Straits and some evo elbows on loads of rides, from trail centres to Yorkshire Moors and Mendips / Quntocks XC stuff. I don't feel out of place, and when I'm firing down a piece of rocky singletrack off the moor at 30mph plus I'm glad I've got them on.
Plus, as the weather gets cooler I find my usual riding shorts + Kyle Straits are plenty warm enough, and I think look loads better than a jey pair of cycling tights, but it's not about how you look so don't get hung up on it.
The knees and elbows won't protect your ribs, well not directly obviously, but I find if I crash wearing my elbow / arm pads then I'll sometimes deliberately break my fall with the bits of me that are padded up.
It's not where you ride, it's how you ride, and if you're likey to crash then wear some protective gear. You wear a helmet and gloves presumably. What's the problem with protecting the rest of yourself?