Try getting out for more than one “proper” ride a week – A few years ago I was a bit unfit and riding with a club once a week would kill me with little appreciable gain. Doing a bit of running (1 or 2 times/week) and gettin out on the bike most weekends now, combined with a 4 mile each way gentle commute, and I’m not remotely killing myself on the weeknight rides or the weekend ones.
Also, don’t push yourself to be at or near the front on rides, just ride at your own pace. There’s no prizes on a social ride!
The 2 mile each way commute you do will be good to keep your legs ticking over, so keep that up.