Are you using an On-One Doofer (the one that tensions down)?
If so then ignore anything else on this thread not related to changing your tensioner for one that tensions UP!
I had exactly your problem and tried everything to fix it, nowt worked until I tensioned UP, then problem solved…
…except after all that I decided I prefered gears.
At least I am now qualified to offer advice on the matter!
If you still have the same problem with an upwards tensioner then start listening to the wise old men with calves of steel above.