Just got back from same trip. Only made it as far as Afan on Friday eve – we drove up to Glycorrwyg because we know from experience you can still get drinks for an hour or two, even if you have arrived hopelessly late. Crashed out in the van and went down to Brechfa for breakfast at the Drop Off bus. From Afan to Brechfa, as mentioned, takes about an hour and fifteen/half – it is quicker to go M4.
Black Lion at Abergorlech (red route at Brechfa) does good food, which we had some of together with rather a lot of beer. Then we nipped back to Afan for a quick go round Whites.
Long and short – Brechfa is def worth the extra mileage, and can be easily combined with Afan.