Yes, and I am talking specifically about the bit between town and Castell Coch, that I used to use. Mostly on weekends, when it was crowded. Although I still think I’d go along the A4054 anyway even if I was going all the way to Ponty.
On the rare occasion I’ve had to work weekends and commuted I’d agree you do get a lot of walkers. They seem to be a different breed too who are totally oblivious to any other trail users. During the commute hours walker seem to be more aware and accommodating.
As for the A4054 v Taff Trail to Pontypridd, if anything the TT surface is better and of course has no traffic to contend with. However, it is about a mile further so takes about 5 minutes longer. But I’ll put up with that not to have to ride in traffic.