The debate isn't about whether oppressing women is wrong, it's about whether the veil is oppressing women.
But it does come down to "knowing what is best" for people. We do it all the time BTW. I made an earlier point that because you can find people who may claim that forced marriage/FGM has been the best thing that ever happened to them (certainly with the latter) I suspect that most people would agree that this is a terrible practice and those that commit it upon their children should be punished. We dont need to analyse their culture to come to that conclusion.
I do think it is a sign of oppresion but it is less overtly obvious to the observer.
In terms of telling people not to wear it or providing a practical solution to what I dont like to see, then it does become more difficult.