I do wash mine (tops and socks for cycling, leggings for under trousers when walking in the winter) but not all that often, as others have said it does work and it means you can pack lighter when on a multi-day trip.
As an experiment a couple of years back I wore a merino top for a week of trekking without washing and at the end of the week it didn't really smell whereas most of the parties synthetics were nasty after a couple of days. On the downside thicker weights do hold a lot of liquid and the cheaper stuff tends to fall apart quicker than manmade fabrics. Sportwool is a nice compromise but starts to smell quicker as a result.