Just curious op, is it a 29er?
My Jeffsy 29er can be a bit how you describe. I’ve only done a limited amount of messing around as I don’t really mind it too much having got used to it. My first 29er you see.
That said,I went from a 60mm stem to a 35mmm initially and it felt very direct but a bit too twitchy for my limited abilities on the downs.
Went to a 50mmm and its a nice compromise. Got higher rise bars to try soon but that’s as much about having gad neurosurgery in my neck as much as anything. I like a sit up and beg riding position.
This is only one variable you ask about though. Forks are 46mm offset incidentally but I’d be a liar if I said I really understood the effects of offset. Particularly when tied into all the other variables.
I suppose the best (limited) advice I can give is swap out one item at a time and ride it a fair bit then make a second change etc if you need/want to. Starting with cheap/free stuff first of course. Spacers then stem blah,blah.
Edit: A lot of what hols said basically.