No, and I was brought up in Annan.
I actually think Dumfries has improved a lot in the last ten/fifteen years, it was a hole back in the early/mid 90’s. Carlisle is a decent small city, if you can stomach the accent and a bit of aggro if you happen to be from North of the Border. Not that far away either and has a train link to Dumfries etc…
I would suggest that if you are moving to the area but have family up north/ down south then being on the Lochmaben/Lockerbie side nearer the M74 might be better for a quick get away, Lochmaben’s quite a nice wee place, Lockerbie too.
Further the other direction Castle Douglas and Kirkcubright are nice.