Where did it break, and can you wear an external support to help in the case of an off etc?
I broke my wrist (end of the distal radius) earlier this year, and was on the turbotrainer immediately (in cast) road bike as soon as cast came off (~5weeks) and went to the alps with The WhiteRoom after approx 8 weeks! I wore a wrist support, more for extra insurance than because i needed to. Had a couple of offs on the alps trip, but all was fine. I’d say use it as early and as much as possible, but CAREFULLY! If you come off, and tear the plating out of the bone it’s going to be V nasty to fix again!
No matter how much the Dr’s tell you, you know yourself how you are feeling, and should be able to make a sensible call as to how much to use it etc. Having it ORIF’d early will have helped dramatically of course.