That's kinda true….
The pic above is a Trent 1000. The blue system (ie a fan, shaft and multiple turbine stages) is the 'LP' system, the yellow system (8 stage compressor, shaft and single stage turbine) is the 'IP' system and the red system (6 stage compressor, shaft and single stage turbine) is the 'HP' system.
The way to think of it is the 'core' of the engine generating all the power is the HP and IP system. This core is used to generate power which the LP turbine extracts to drive the fan.
At idle the Fan will only rotate at around say 10% of it's full speed (say 3000 rpm), generating only a few thousands pounds of thrust. But as you said, to do this the core shafts, in particular the HP shaft will need to go at 70-80% of its maximum speed (around 13,000 rpm). In going to full thrust (say 75,000lb certified sea level thrust for Trent 1000) the HP shaft will only increase in a bit to get to 100% speed, but the fan will go around 10 times faster!
The eagle eyed among you may have also spotted that in my first post it should be HP6 rather than HP3….
Cheers,
Col.