It’s easy to say but it can be rider tension. Try to relax. Maybe get used to descending on the hoods before going for drops.
Having said that check mechanicals. As noted tyres seated ok, headset. Too tight can be worse than loose. As you ride you’re constantly making subconscious corrections. If the headset is too tight or even brinelled your micro adjustments have to become bigger especially at speed where gyroscopic effect adds to force needed for corrections. Once this starts then you get speed wobble/tank slapper. On motorbikes they recooend that you let go of bars. Oh yeah.
Do your wheels need balancing? Try spinning them with your brakes loosened so they aren’t effecting the spin. If they always stop at same place, typically valve at bottom. Try adding some lead foil on opposite side spokes and spin till stopping point is random.