From 2 to 3 the big difference is the Retina display, but there’s also a better back-facing camera. 3 to 4 is a CPU/GPU enhancement and the Lightning connector, plus a better front-facing camera.
The Retina display is a massive benefit if you do much reading (web, books, whatever) but perhaps less so if you’re just gaming or looking at pictures or watching video. But it’s certainly nice to have whatever you’re doing.
The 3 is cropping up occasionally at HMV and Apple refurbs at great prices if you’re willing to keep a close eye on when stock is in. If you’re buying new at Apple RRP then I’d definitely go for the 4 over the 2. I’ve traded my 3 in for the 4 and I’m hoping that’s going to fix the slight lag in drawing apps that the 3 had.