3 ms·

I wasn't there there, but I remember when someone at gitlab deleted the gitlab.com production database. The team at the time setup a public Google doc with all

by rudasn 3y ago

I wasn't there there, but I remember when someone at gitlab deleted the gitlab.com production database. The team at the time setup a public Google doc with all the info and a public video stream on YouTube with the devs taking about what to do/what they are doing. I've never seen anything like that before or after, although I bet lots of prod db get deleted every day:p

_aaed 3y ago
https://www.youtube.com/watch?v=tLdRBsuvVKc https://www.youtube.com/watch?v=tLdRBsuvVKc This video goes through the timeline of the main points
jszymborski 3y ago
> I've never seen anything like that before or after, although I bet lots of prod db get deleted every day:p Absolutely. I have at least three stories from friends who witnessed these sort of events first hand, and so many of posts in the genre online affirm that its a regular occurence. I've yet to hear of a case where there weren't backups for prod, but I'm sure those exist to.
renegade-otter 3y ago
This happened in the early days of GitHub: https://github.blog/2010-11-15-today-s-outage/ https://github.blog/2010-11-15-today-s-outage/ These sloppy setups are common even today, where it's fairly easy to allow for a simple config error or a wrong environment variable (env=PROD) to wipe out production from a test suite. These safeguards need to be in place in the first days of your codebase.
- vinniepukh 3y ago
  can you an example of those kinds of safeguards? asking for a friend :)
  - conceptme 3y ago
    Not letting devs access production :)
  - renegade-otter 3y ago
    I think one thing is to make sure that when your test suite executes - you should force environment to "TEST" programmatically, before any bootstrapping code even runs. So even if you allow for connection to production (sometimes that's allowed for debugging hard-to-reproduce issues), you want to remove any room for human error. And your code should obviously NEVER be able to connect to anything other than the local test DB when your environment is set to "TEST".
- rudasn 3y ago
  Oh I was under the impression that it was a forgotten reverse ssh tunnel, that proxied localhost:some_port to the production setup. Or maybe that was another story hehe.

by rudasn 3y ago

_aaed 3y ago

https://www.youtube.com/watch?v=tLdRBsuvVKc https://www.youtube.com/watch?v=tLdRBsuvVKc This video goes through the timeline of the main points

jszymborski 3y ago

> I've never seen anything like that before or after, although I bet lots of prod db get deleted every day:p Absolutely. I have at least three stories from friends who witnessed these sort of events first hand, and so many of posts in the genre online affirm that its a regular occurence. I've yet to hear of a case where there weren't backups for prod, but I'm sure those exist to.

renegade-otter 3y ago

This happened in the early days of GitHub: https://github.blog/2010-11-15-today-s-outage/ https://github.blog/2010-11-15-today-s-outage/ These sloppy setups are common even today, where it's fairly easy to allow for a simple config error or a wrong environment variable (env=PROD) to wipe out production from a test suite. These safeguards need to be in place in the first days of your codebase.

vinniepukh 3y ago
can you an example of those kinds of safeguards? asking for a friend :)
- conceptme 3y ago
  Not letting devs access production :)
- renegade-otter 3y ago
  I think one thing is to make sure that when your test suite executes - you should force environment to "TEST" programmatically, before any bootstrapping code even runs. So even if you allow for connection to production (sometimes that's allowed for debugging hard-to-reproduce issues), you want to remove any room for human error. And your code should obviously NEVER be able to connect to anything other than the local test DB when your environment is set to "TEST".
rudasn 3y ago
Oh I was under the impression that it was a forgotten reverse ssh tunnel, that proxied localhost:some_port to the production setup. Or maybe that was another story hehe.