We previously had…brake levers too small to comfortably support your hands, so it was more comfortable to ride on the drops
That is simply not true. 30 odd years ago riding on the hoods was the norm, and there was only one person in our club who rode on the drops most of the time.
we also had adjustable height quill stems and saddle to bar drop wasn’t as big.
It is not as simple as that. For many pros the levers themselves were lower by virtue of typically being mounted furthest forward on the curve, which meant that the top and the ramps were higher (giving a wider – arguably better – range of different positions than modern bars with the hoods, ramps and tops all at the same level). Non racers tended to rotate the levers up closer to the ramps.
Certainly the shape of bars has evolved in conjunction with other changes to the ergonomics of the bike and STI/Ergolever shape, and deep drop bars like Cinelli Model 66 used by Lemond and (I think) Merckx have virtually disappeared, as has the use on the road (as opposed to the track) of the type like Cinelli Model 65 used by Sean Kelly which was supposed to be preferred by sprinters because the curve started further in on the tops, resulting in the ramps being less in the way of the forearms when sprinting with hands on the drops. The fact that modern sprinters don’t require similar bars probably just goes to show that a lot of this was convention and fashion, and that’s probably just as true today (except where a change has been driven by rigorous scientific tests, such as in a wind tunnel).