Have you tried turning all the radio stuff off on your current phone to see how long the battery lasts when all it’s doing is GPS? If you have wifi on all the time it will be wasting power looking for a network, and the actual phone connection turns up the transmit power when it’s having trouble finding a cell tower, so if you ride into any bad signal areas (or it’s in your pack next to your foil-wrapped sandwiches!) it will use quite a bit of power doing that. If you put it in flight mode that should turn off all the bits that transmit, which is what uses the most power.
Of course you won’t receive incoming calls that way, but I have mine on silent mode anyway while I’m riding as I certainly wouldn’t stop to answer it! Your main issue then will be the screen, as you’ll presumably want it on most/all of the time to see the map, and the screen eats up power too, especially if it’s set to max brightness which you might need for outdoor use. If you could live with just turning it on when you actually want to look at the map then that would help a lot. The only way to be sure how much all that would help is to try it and see how long your battery lasts.
Another option is to carry an external USB battery pack that plugs into the phone and feeds it power (assuming your phone charges from USB). I’ve used them before when away from civilisation, and you can get them either with a rechargeable battery built in or to take normal batteries such as AA or AAA so you can carry spares if you find you still run out, though I doubt you’d run one set flat and flatten the phone battery in one ride. The only thing with that is they’re not usually very rugged so I don’t know how well the USB connection into the phone would stand up to being knocked about on the trail and you’d need to make sure the whole thing is waterproof.
Or just buy a Garmin! But you’re right that they’re generally not cheap and often don’t come with the map you might actually want to use.