All machines (can) run the same services, and we roll an entirely new self contained OS image for every update. That goes through multiple expanding levels of

All machines (can) run the same services, and we roll an entirely new self contained OS image for every update. That goes through multiple expanding levels of canaries prior to a fleet wide rollout. Bugs that escape the canaries are quite rare. One of the last ones I recall was a release that contained a firmware update for a NIC that had a super rare bug that would lock up the NIC, and we only started to see it when a sizable percent of the fleet was running that firmware.