I have quite gone off them, in spite of their being a daily (well, mon to fri) staple of my work wardrobe.
A smart casual "compromise":
1. Jean cut corduroys – not all loose and baggy like some M&S Christmas cast-off, mind. Avoid chinos unless you can help it – they start the day nicely pressed looking reasonably smart, and soon become baggy and creased, making you look like a disgraced geography teacher.
2. Leather shoes – not the sort you'd polish, or consider smart enough for a suit. So, Oxfords are out, casual brogues are in.
3. Open necked shirt – needs to be a "casual" cut – i.e. not the sort of shirt you would wear with a tie.
4. Sewater – fine material, not a chunky knit, and no patterns/logos (Ralph Lauren's Polo logo is just about acceptable, IMO)
I'd try to avoid a jacket. You need something really well cut – not like Clarkson's – to cary this off in a "casual" way. TJ isn't fasr off with the suggestion of a tweed, but it would be overkill for a team building event.