5 ms·

Just wondering (sort of—I'm also slightly trying to point out that you're being a bit of an armchair Reddit ops person :) What makes you think that the issue R

• paxys 7y ago
  I definitely am (but we are on Hacker News after all). Without knowing anything about this particular issue, at a basic level I'd think any non-trivial site should have a system of rolling out changes to a small percentage of servers and have automated monitoring that fires alerts if it notices any disruptions in functionality. If nothing goes wrong then gradually roll out to 100%. If something does then stop the rollout and go back to the last stable version. Then engineers can look at what went wrong.
  • hobofan 7y ago
    From my experience that mostly only applies to the application layer. Especially updates on distributed databases are often not as easy as "stop and rollback" as you have to achieve a stable quorum or other similar dynamic states like that which you can't easily achieve from restoring a snapshot. There your best shot often is to learn how to deal with all the failure scenarios manually and do as many tests on a staging system until you feel comfortable.