I was in exactly the same position as you OP. I took the risk, after all it was only a litle stream really.
Fast forward 3 years and a number of close shaves and I couldn’t sleep whenever there was a heavy rain alert. I spent nights clicking refresh on the met office rain radar. Eventually I convinced my wife to move. It was that or have me committed!
She agreed. We moved in the February. The house flooded the next December.
It wasn’t the flooding event I was scared of, it was the upset to the family. Six months out of home. Nowhere that I could rent with the dog and cats. The loss of resale value (which is ultimately my kids inheritance). All that lot. Despite the fact I wasn’t there when it flodded I wish I’d trusted my instict and walked away.
I live on top of a hill now.