3 ms·

development against production? no, no, no. What happens when you're testing your delete code? oops. The complexity of multiple environments has increased, but

development against production? no, no, no. What happens when you're testing your delete code? oops. The complexity of multiple environments has increased, but thats no excuse to develop on production.

withinboredom 4y ago
It was a deliberate decision. Most bugs and challenges were due to scale (that you cannot replicate in a synthetic environment), and due to global distribution, not due to bad code. And yeah, in the 5ish years I was there, I saw someone delete a database table in production exactly once. It took ~1 hour to finish deleting (hundreds of millions of rows takes awhile to delete), and it was back online seconds after that from a backup. I think we lost virtually no data. We just disabled the feature until the data came back. I can't imagine that happening nearly as flawlessly at any company before or since.
- jpmoral 4y ago
  Did the recovery go well because of the single-env set-up, or despite of it?
  - withinboredom 4y ago
    Well, the fact that we could just 'turn the feature off' was something that you'd generally only see because you /must/ deploy code that is disabled to keep PR's relatively small. So pretty much anything and everything can be entirely disabled -- it really depends on how long the feature has been enabled though. If it has been 'always on' for years, its possible new devs didn't include the check when modifying/adding the feature to new places. Then there was the fact that it was deleted at all. Once in 5 years, with a total dev team in the hundreds ... isn't that bad. They only pulled it off by doing something they were told not to do. That can happen anywhere. Also, since devops knew that it /could/ happen (because there is always someone who does what they aren't supposed to do -- including themselves when things break) took streaming backups of data. They were able to rebuild the table while the original was deleting and swap it back in as soon as the db released its write-lock. To me, I hadn't seen that level of 'fixing things' in prod. At 'regular' companies, if someone managed to delete a table, they'd probably spend at least twice that time finding someone who knew where the backups were and figuring out how to restore them. Meanwhile prod would probably be down completely since they can't disable the feature.
quanticle 4y ago
Testing delete code against production isn't that big of a deal. If you screw up, the "oops" is immediate and obvious, as is the recovery strategy — restore from backup. The real fear with deploying code directly against production is data corruption. I've seen more than one instance where two pieces of code, both of which had impeccable unit test coverage, interacted in strange ways to cause data corruption, because of mistaken assumptions that their respective authors had about each others' code. This is the kind of thing that you need integration tests to catch. Integration tests need some kind of common environment (even if it is ephemeral) in which to run. And now you have a development environment.
- chadcmulligan 4y ago
  lol, I didn't enumerate all the ways this is bad. Restoring from backup can mean an outage and possible loss of data (the intervening transactions that are lost), this is to be avoided, it also assumes your backups work, often an untested case. Corruption is bad to, the entire idea is bad.

withinboredom 4y ago

It was a deliberate decision. Most bugs and challenges were due to scale (that you cannot replicate in a synthetic environment), and due to global distribution, not due to bad code. And yeah, in the 5ish years I was there, I saw someone delete a database table in production exactly once. It took ~1 hour to finish deleting (hundreds of millions of rows takes awhile to delete), and it was back online seconds after that from a backup. I think we lost virtually no data. We just disabled the feature until the data came back. I can't imagine that happening nearly as flawlessly at any company before or since.

jpmoral 4y ago
Did the recovery go well because of the single-env set-up, or despite of it?
- withinboredom 4y ago
  Well, the fact that we could just 'turn the feature off' was something that you'd generally only see because you /must/ deploy code that is disabled to keep PR's relatively small. So pretty much anything and everything can be entirely disabled -- it really depends on how long the feature has been enabled though. If it has been 'always on' for years, its possible new devs didn't include the check when modifying/adding the feature to new places. Then there was the fact that it was deleted at all. Once in 5 years, with a total dev team in the hundreds ... isn't that bad. They only pulled it off by doing something they were told not to do. That can happen anywhere. Also, since devops knew that it /could/ happen (because there is always someone who does what they aren't supposed to do -- including themselves when things break) took streaming backups of data. They were able to rebuild the table while the original was deleting and swap it back in as soon as the db released its write-lock. To me, I hadn't seen that level of 'fixing things' in prod. At 'regular' companies, if someone managed to delete a table, they'd probably spend at least twice that time finding someone who knew where the backups were and figuring out how to restore them. Meanwhile prod would probably be down completely since they can't disable the feature.

quanticle 4y ago

Testing delete code against production isn't that big of a deal. If you screw up, the "oops" is immediate and obvious, as is the recovery strategy — restore from backup. The real fear with deploying code directly against production is data corruption. I've seen more than one instance where two pieces of code, both of which had impeccable unit test coverage, interacted in strange ways to cause data corruption, because of mistaken assumptions that their respective authors had about each others' code. This is the kind of thing that you need integration tests to catch. Integration tests need some kind of common environment (even if it is ephemeral) in which to run. And now you have a development environment.

chadcmulligan 4y ago
lol, I didn't enumerate all the ways this is bad. Restoring from backup can mean an outage and possible loss of data (the intervening transactions that are lost), this is to be avoided, it also assumes your backups work, often an untested case. Corruption is bad to, the entire idea is bad.