I’ve hit 57mph down a hill once (with panniers so probably similar weight distribution to you) – I did have fairly wide slicks and suspension forks on which may have helped. I didn’t experience any wobbles, but didn’t have reflectors fitted – I can imagine they might not help based on the wobble I get from one of my bikes on the stand.
I wussed out of going for the big 60 due to a slight bend, not great road surface and a car coming the other way. Bikes should be OK up to about 60-70, you can hit that with a suitable hill. Is the wobble going away above 40mph or getting worse?