I've done it (over several weeks), and it's doable.
However, I wouldn't do it again. Panniers make life a hell of a lot easier.
Careful strapping of bags on will ensure they don't get caught places and don't fall off, but there's always some faff involved with keeping them that way.
It's also a major hassle putting them on, and taking them off – which might not seem like a big thing, but getting bags off into youth hostels, or for packing on and off trains. It's muchly frustrating.
I'd have a look at some of the panniers that will convert into rucksacks, if that's a strong point for you.