I broke it when I was a teenager after getting in a drunken argument and punching a metal bin.... anyway, it looked like I had half a tennis ball under my skin, it was that swollen. Girlfriend at the time was a training nurse and her exact words were "you havent broken it"
I ignored her and went to A&E and it was a broken scaphoid. I was told that the scaphoid is a horrible break because it gets little oxygenated blood and takes ages to recover. I continued working and driving on mine but took it easy. I was in plaster a total of 3 months and as infuriatingly annoying it was I am glad because I have had NO issues since. Upon research it seems people dont give it enough time to heal and have all kinds of problems. They then get it operated on which generally makes the problem worse.
I went out to Morzine DHing in the alps for 2 weeks just a week after my cast came off and had no problems with it bar being a bit weak in that arm for a little while. I get no pain, no clicking, no disformity or strength issue. It doesnt ache with long days riding and I put that down to just letting it take its time.
I encourage you to keep the cast on longer, I am no doctor and I can only go from my own experience but I think 4 weeks isnt long enough. Im sure others will agree.