If the refund was incorrect then I’m pretty sure that they can ask for it back. I think there may be a six year limit on tax enquiries.
You need to work out if you’ve paid the right amount of tax over the rest of the year. There are websites that will tell you how much you should pay given your salary, e.g. http://www.incometaxcalculator.org.uk and you can compare the number at the bottom of that with your take home pay. If your actual tax home is significantly lower that’d explain why you’re getting refunds.
If you are on the wrong tax code, you could be paying too much at the end of each month, which is why you’re getting the refunds, although normally they would issue a new tax code to fix it.