I leave them on all year on both vehicles. Not really for any reason than they both were due for new tyres at the onset of winter so winter tyres are what went on and the tyres I had were't worth keeping.
Depends where you are in the UK but as with Mr Scotroutes Its only been a fairly short part of the year where the tyres have been outside their 'zone'. They're best under 7 deg and measurably worse above 20 deg - well over 20deg at the times of day when I'm actually behind the wheel hasn't really happened very much.
This is with the caveat that I don't use fashion tyres, they're not big/wide/low profile/ expensive so my tyres are unfashionable and relatively cheap, and I don't do high milages (particularly as my driving is split between two vehicles) so if there is concern about accelerated wear its not enough of concern for me to bother with.