It doesn’t take me that long to download games, more 30 minutes to hour for a full size +30gb-ish game. I’ve found the MS servers pretty quick. I did have issues with the really slow WiFi adapter on the old 360 but cured by using Ethernet cable.
The update issue is a real pain though. There are monthly OS updates, the bigger OS updates can mean updates for all games, and any game with online elements there are regular updates often to balance on line elements. The way to avoid the waits is to not use power save shut down, but the “sleep” mode and the Xbox will update everything when switched “off”. Not ideal and you need to watch data download volumes if you don’t have unlimited