You’re not too far from Woodbury Common there which has a lot to offer, but also as I understand it there are a couple of very good loops that you can string together around Sidmouth/Ottery St Mary, with a fairly extensive bridleway network through some challenging Devonshire hills to the North East of Sidmouth..
I certainly wouldn’t advocate riding the South West coast path in that area, especially as it has now been designated a UNESCO world heritage site.. But there are some stunning singletrack walking trails on that path.. The possibility of conflict with walkers is moderate near to built up areas, but less so as you venture further out, especially if you abide by the code of ‘don’t be a dick’