I’ve just had the same issue. I couldn’t remove the play by tightening so I added a thick rubber washer before reinstalling – it removed most of the lateral play but it still shifted very poorly despite hours of fettling.
Never was a fan of SRAM, so ordered a new SLX shifter and rear mech for £70….faultless ride yesterday :-)