By the way, I'm neither a particularly a skillfull rider, nor a lightweight rider, but I'm under no illusion that a lighter bike will improve my speed or overall ability
That's not what I am saying. Most of us are not racers, so a few seconds saved is of no consequence.
However it generally FEELS nicer and more fun to ride a more responsive quicker handling bike.
When you are zipping through singletrack your bike moves this way and that pretty quickly, and it being lighter helps it do this. A lighter bike also responds more quickly in that first quarter of a pedal stroke when you stamp on the pedals.
If you've never tried a 21lb bike, I suggest you do so - you'll see what I mean! You may not like it overall, to be that light a bike generally has to be XC race spec or thereabouts, which is less appropriate for throwing off big things if that's what you do.
However if you have a 35lb Orange 5 say, and you compared it with my 27lb one, you'd see a big difference on singletrack and on climbs. Technical climbs by the way are about manhandling the bike a lot too, and lighter weight helps there. Of course, on the way down on big rocky stuff my 5 would not be as good. The forks are 140mm Revs and the 90mm stem is a compromise, because I like to ride up too.