4 ms·

Yeah. “We rm -rf on production server, and our backups are useless, but we’re public and honest!” Sorry, not impressed.

by oefrha 7y ago

andyroid 7y ago
This happens everywhere. You just don’t know about it precisely because companies are normally not public and honest about it.
- nl 7y ago
  It really doesn't happen everywhere. Most places with decent devopss hygiene have defense-in-depth around their backups. I've heard of people dropping production databases in big companies (but saved by backups). There are some stories around the bitlocker blackmail thing that had similar impact, but that was with a malicious opponent. The only thing similar I've heard for the notorious self modifying MIT program (for geo-political coding) in the 1990s which destroyed itself without backups.
  - icebraining 7y ago
    Gitlab was "saved by backups" as well. They lost some data since the latest backup, which is rather common.
  - vidarh 7y ago
    Most places don't have decent "devops hygiene".
- oefrha 7y ago
  > You just don’t know about it precisely because companies are normally not public and honest about it. If a big company lost a ton of user data, I'd absolutely know about it, whether they have Apple-level secrecy or not.
  - andyroid 7y ago
    The incident described did not result in loss of tons of user data, and neither will most incidents, whether you choose to be open about them or not.
    - oefrha 7y ago
      What are you talking about? > This incident caused the GitLab.com service to be unavailable for many hours. We also lost some production data that we were eventually unable to recover. Specifically, we lost modifications to database data such as projects, comments, user accounts, issues and snippets, that took place between 17:20 and 00:00 UTC on January 31. Our best estimate is that it affected roughly 5,000 projects, 5,000 comments and 700 new user accounts. https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/blog/2017/02/10/postmortem-of-datab... Yes, most incidents from most companies don’t result in this kind of data loss, which is why GitLab stood out.
      - icebraining 7y ago
        How do you know what most incidents result in? For example, when Github deleted their production database[1], they simply gave no numbers of affected users/repositories. We do know that the platform already had over 1M repositories[2], so 5000 affected seems perfectly possible, but their lack of transparency protected them against such claims. And that lack of transparency seems to me to be the norm. [1] https://github.blog/2010-11-15-today-s-outage/ https://github.blog/2010-11-15-today-s-outage/ [2] https://github.blog/2010-07-25-one-million-repositories/ https://github.blog/2010-07-25-one-million-repositories/
  - exikyut 7y ago
    MySpace lost all its music from 2003 to 2015: https://news.ycombinator.com/item?id=19417640 https://news.ycombinator.com/item?id=19417640 Probably a few hundred TB or so. Maybe nearly a petabyte?
    - oefrha 7y ago

Yeah. “We rm -rf on production server, and our backups are useless, but we’re public and honest!” Sorry, not impressed.

by oefrha 7y ago

Yeah. “We rm -rf on production server, and our backups are useless, but we’re public and honest!” Sorry, not impressed.

This happens everywhere. You just don’t know about it precisely because companies are normally not public and honest about it.

nl 7y ago
It really doesn't happen everywhere. Most places with decent devopss hygiene have defense-in-depth around their backups. I've heard of people dropping production databases in big companies (but saved by backups). There are some stories around the bitlocker blackmail thing that had similar impact, but that was with a malicious opponent. The only thing similar I've heard for the notorious self modifying MIT program (for geo-political coding) in the 1990s which destroyed itself without backups.
- icebraining 7y ago
  Gitlab was "saved by backups" as well. They lost some data since the latest backup, which is rather common.
- vidarh 7y ago
  Most places don't have decent "devops hygiene".
oefrha 7y ago
> You just don’t know about it precisely because companies are normally not public and honest about it. If a big company lost a ton of user data, I'd absolutely know about it, whether they have Apple-level secrecy or not.
- andyroid 7y ago
  The incident described did not result in loss of tons of user data, and neither will most incidents, whether you choose to be open about them or not.
  - oefrha 7y ago
    What are you talking about? > This incident caused the GitLab.com service to be unavailable for many hours. We also lost some production data that we were eventually unable to recover. Specifically, we lost modifications to database data such as projects, comments, user accounts, issues and snippets, that took place between 17:20 and 00:00 UTC on January 31. Our best estimate is that it affected roughly 5,000 projects, 5,000 comments and 700 new user accounts. https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/blog/2017/02/10/postmortem-of-datab... Yes, most incidents from most companies don’t result in this kind of data loss, which is why GitLab stood out.
    - icebraining 7y ago
      How do you know what most incidents result in? For example, when Github deleted their production database[1], they simply gave no numbers of affected users/repositories. We do know that the platform already had over 1M repositories[2], so 5000 affected seems perfectly possible, but their lack of transparency protected them against such claims. And that lack of transparency seems to me to be the norm. [1] https://github.blog/2010-11-15-today-s-outage/ https://github.blog/2010-11-15-today-s-outage/ [2] https://github.blog/2010-07-25-one-million-repositories/ https://github.blog/2010-07-25-one-million-repositories/
- exikyut 7y ago
  MySpace lost all its music from 2003 to 2015: https://news.ycombinator.com/item?id=19417640 https://news.ycombinator.com/item?id=19417640 Probably a few hundred TB or so. Maybe nearly a petabyte?
  - oefrha 7y ago