As someone who has had the job of destroying old helmets at outdoor centres, I hope I can add something useful.
We would throw out (polystyrene) helmets regardless of condition at 2 years (3 for proper 'hardshell' climbing or canoe helmets). If damaged before then, just immediate bin. Bear in mind, these helmets were NEVER treated properly - dropped, shoved under minibus sears, lots of falls/bangs etc, and certainly worse than you or I would treat our own helmet.
When throwing out, being 'safety critical', we had to destroy them - cue lump hammer time. Most of the helmets were very strong still - took a good hard blow from a hammer to destroy them. Usually you would get a big dent and a crack up one side. BUT, every set we chucked had at least one helmet that just disintegrated / was much easier to smash / separated into a couple of bits much more easily.
I couldn't tell you the reasons for this, just that in every bunch, one or two were 'duff' / damaged / weak.
My own helmet I change about every three years, or if damaged before. My last helmet was a year old, I thought was fine, no major bumps/offs (or that I can remember), and yet putting the washed pads in one night I discovered the polystyrene had cracked from one side to the other - and was basically held in place by the outer shell.
I think its a case of inspect it, keep it carefully (out of UV rays importantly) and if it gets beyond 'a lot' of use, change it. The risk is small, but it is there.