There is variation and I bought some xtr pedals largely for this reason (very low bb in this case, so every little helped), but they’re only 1mm less than XT in height over axle so I think you’re talking small amounts. That said, when I swapped them for flats, for messing about on holiday in flip flops, I only had to drop the saddle a little bit, like much less than 10mm I guess.
What clip shoes? I find xc ones to feel a lot taller than my AM9s or other flat-style shoes, although not compared back to back.
I guess the thinnest flat pedals will always be a long way off the thinnest SPDs because of the mechanism, but I think you can get your SPD setup within reach of the average flat.