Im pretty sure it will wear the frame out eventually if I leave it like it is.
It will!
My brother has a Marin that this has happenned to.
The top hats have worn the frame to the point they can no longer be tightened up properly, even with new bushes and top hats there is permanent play, he now needs slightly wider than standard bushes.
I’d fit a Fox low friction kit, I don’t think they can sieze the way alloy ones can.