Ross: leave the spacers out and boly your slx straight onto your mounts.
I have done exactly what you are asking on a couple of bikes (shimano brakes onto avid mounts) and they were just fine. Avid calipers are ‘spaced’ so that you need the conical spacers to set them right whatever mounts (and matching rotor size) you use, ie the difference is on the caliper not the adapter.
(The difficulty you may have later down the line is that shimano do not make a 185mm rotor. So if you replace your worn out or bent rotors with shimano 180mm, you will need a new front mount too as your brake will be 2.5mm too far away from the centre of the rotor. This becomes dangerous when you wear most of the pads except 2.5mm at the top and then those two ‘steps’ close against each other and the rest of the pads stop gripping the rotor and slowing you down.)