I am not sure if by downshift you mean changing down to a smaller gear, or dropping the mech down to a smaller sprocket.
If the latter, I find that if the brake levers are squeezed slightly, then the smaller shift lever can fail to engage (are you riding on the hoods and unconciously pressing the brake levers when changing?)
I think I’ve noticed a similar effect without pressing the brake levers, if I push the smaller shift lever in a slightly diagonal direction (inwards and towards me, rather than just inwards).
However, I usually realise what I have done when this has happened, and either take pressure off the brake levers or push the smaller shift lever just inwards, so it doesn’t take 3 or 4 attempts to properly engage.