I’m surpised you felt much difference going from 70 to 65.
Anyhow.
I recently went from a 70 to a 50 mm stem.
If you go for 55mm this will mean the bike will feel shorter.
You’ll need to adopt a slightly different position to get weight over the front for climbing.
Conversely, going downhill may feel more stable.
The steering will feel quicker and more twitchy, which takes a bit of getting used to – there are pros and cons to this… depends on personal preference
The only way you’re going to know if a shorter stem is a good idea is to suck it and see.