The zoo is great. My 3 and 1 year olds both love it and there’s plenty for five year olds as well. The Botanical gardens are also good fun and free (apart from the greenhouses) but the cafe tends to be quite crowded.
The museum is also great for kids, especially if it’s raining.
TBH the castle is not that friendly to kids. Last time I went it struck me as a fairly poorly presented museum, and the kids I was with (cousins) got bored pretty quickly. Also there’s a lot of hilly walking once you’re inside the castle.
Princes Street gardens are also not that accessible just now. The only access is at either end because of the tram works, which also makes driving in the city an awful experience. Use the park and rides, which are plentiful.