I used to carry an SLR around on the road bike. It’s a lot of weight to carry though, especially if you spend a lot of money on lightweight components and starve your belly of any favourite treats (I don’t really do either, so it didn’t bother me too much!)
However, because my road rides became more purposeful in terms of riding, I found myself using it less and less, to the point where I hardly used it at all, and so I’ve just bought myself a Lumix LX-5. Which is probably about as small as you’ll get without going into budget cameras.
Haven’t tried it in a jersey pocket as I’m a little concerned about it bouncing out. So I’m still figuring out the best way of carrying it…without using my SLR bag!