3 ms·
Do you run all service on the new kernel or is there a roll out process where some services are canaries? I have one service that is low traffic that I use to
by hinkley 5y ago
Do you run all service on the new kernel or is there a roll out process where some services are canaries?
I have one service that is low traffic that I use to test out new base images and core api changes. It’s easy to generate synthetic traffic to see how it behaves, as a lot of its traffic comes from the build process of more complex services.
- drewg123 5y agoAll machines (can) run the same services, and we roll an entirely new self contained OS image for every update. That goes through multiple expanding levels of canaries prior to a fleet wide rollout. Bugs that escape the canaries are quite rare. One of the last ones I recall was a release that contained a firmware update for a NIC that had a super rare bug that would lock up the NIC, and we only started to see it when a sizable percent of the fleet was running that firmware.