Stayed in Dyffryn, on one of the campsites, with two small kids. The following year we stayed in Barmouth.
The beach at Dyffryn is fantastic, although there's not much to the village itself.
Handy for the Pont Scethin ride though.
Barmouth was better with the kids, as they were old enough to ride the Mawddach Trail, which takes you over Barmouth Bridge to Dolgellau. There is a nice pub on the Estuary just outside Dolgellau – King George I think it's called, which was ideal for a lunch stop before heading back.
A four-year-old could do the Mawddach Trail, as it's completely flat. My daughter was five the first time she did it.
And your only 25 minutes from Coed-y-Brenin.
We were going to go back this year, but I've managed to talk the other half into going to Morzine instead.