We were in the Gower last Easter and I had a lovely ride doing all of the (non-footpath) bits of the Gower Way. From the car park in Rhosilli (or go back to the coastguard lookout above Worms Head if you're a completist) head back inland along the road for a couple of hundred yards, right at the church then on and turn left at the phone box (Middleton) Follow this road which becomes a lane past Fernhill farm on R. From here it swings R to follow fields through Kingshall and old Henllys farms, bringing you out in LLandewi. Use the road for this bit as the next bit of Gower Way is footpath and the ground is boggy anyway. Turn left up the road and follow it round to Burryalley, then left to Hillend(not the Hillend on the end of Rhosilli Down). From here you've got the whole of the central ridge of the Gower ahead across to Penmaen. Views from the top are fantastic.
Other than that, then as crispedwheel says there are a pair of bridleways across Rhosilli moor itself on the sea side of the ridge, starting from the path behind the church, and another on the land side which meets up with them at Hillend, it starts along the lane to Fernhill farm, as above. Enjoy.