To be honest quite a bit of bodyweight stuff I would count as “heavy”, dips/pullups/etc and probably a lot of stuff gymnasts do. The human body has a fair amount of mass to it. As for swimmers, well here is a quote from Phelps:
“In preparation for Beijing, I started adding weightlifting to my dry-land work,” Phelps says. “Since then, we’ve expanded the amount of weights I am using, and I’m running more than I ever have. Pushups and pullups have also always been essential.”
Certainly the swimmers I see in the pool who don’t also lift do not look particularly muscular. I would be very surprised if most sports people who look muscular (including gymnasts) get that way without lifting some heavy weights.