Plenty of great castles noted above, but, being south Wales, we have tonnes of them around here. I can ride past two on my commute home tonight, and, if I wanted a longer ride, could add the ruins of another 4 or 5 on Gower that I can think of off the top of my head.
I will suggest an oddity that I have a soft spot for – Penlle’r Castell.
http://www.ecastles.co.uk/penller.html
Nobody actually knows what it was, but probably the northernmost fortification of the Lordship of Gower. To get there on a bike you have some long, nasty climbing across barren hills, then a short grassy track over the moor, and then you find these holes in the ground. It’s a random place, but has fantastic view north – the Amman valley below, windfarms to the left, and the last time I was there a massive black storm approaching from the right. Darkness in the east, the sunsetting in the west – it looked very dramatic. It would be a good bikepacking spot, I’ve decided.