Common wisdom these days is that wider-and-softer is faster/easier on rougher surfaces, which is fairly easily explained.
I still find 23s (on a nice light bike at least) give a psychological sense of urgency that 28s don’t, though: the faster you pedal, the better you skim across the surface; whereas larger tyres, being more comfortable at lower speeds, give you less incentive to drop the hammer. So they end up making me faster even if they’re theoretically inferior. YMMV. (I’ve not tried the compromise 25, though my 28s seem to come up as 25 anyway.)
And personally I don’t think the weight difference is a big deal unless you go over 28.