It’s not just stupid cats who get lost. Cats are deeply territorial and once their territory has been established they will know every inch of it. A cat in a strange territory will be extremely nervous and can get freaked out by the sudden appearance of an unknown cat/dog/human causing to take off and travel much further than it might otherwise. It might then struggle to know the way back or be too nervous to attempt it, fearing that the thing which spooked him would still be lurking/threatening.
Cats rarely get lost but it can happen. Although only really in strange territory. I would let it get fully used to the new house before letting out, and yes, a few days doesn’t seem enough, there’s no point taking risks imo. When you let it out reduce the possibility that it will wander too far (you only want it to sniff around the garden at first) by making sure that you let it out hungry, early in the day (gives you more time to look for him if he wanders, and cats feel much braver at night – increasing the chance they will wander) and preferably when there’s poor weather.
IMO