Remove your spacers and see what it feels like.
IMO 10mm makes a huge difference.
You could offset the lower stem by a larger riser bar or stem with steep angle perhaps?
in my opinion though, 185mm is too short. Its nice to have a long steerer and it is (I think) ok to have spacers above your stem st it is nice to have the opportunity for flexability to move your stem up or down a bit.
Short steerers will also limit the available customers when you flog the fork second hand.
I bought a new set of handlebars the other day and they came with a 40mm stem. The bike was unrideable and it felt like I was steering a Sinclair C5. 70mm stem later and its superb. That was an out jutting length of just 30mm more. A small amount on the bars stem etc adds up to feel like a lot.