3 ms·

They forgot one: Natural disaster deleted all the data and there was no backup. I was a paying subscriber to a great little site called magweb ~20 years ago. T

by livrem 3y ago

They forgot one: Natural disaster deleted all the data and there was no backup. I was a paying subscriber to a great little site called magweb ~20 years ago. They scanned old and new (military) history and (war) game magazines and posted HTML versions, with permission of original publishers. It was really nice and explicitly allowed users to print or save copies of articles. No DRM. Perfect example of how a site for reading magazines online should work. Everything just basic HTML that even worked great to read in phone browsers 20 years ago. Then Hurricane Katrina hit and the server that was apparently running out of the owner's basement somewhere in that area was flooded, with no working backups. I still have not found any traces of most of the 40000+ articles, other than the few I had saved, that used to be on that site. Since it was paywalled only a tiny part of the site is available on the wayback machine. https://web.archive.org/web/20050529083811/http://www.magweb.com/ https://web.archive.org/web/20050529083811/http://www.magweb... (No, I can't imagine running a business and scanning magazines for 9+ years and not make sure to have backups of everything.)

Hamuko 3y ago
There was also the OVHcloud data centre fire in 2021, although it's debatable if it'd fall under the same "natural disaster" category. https://www.datacenterdynamics.com/en/news/ovhcloud-ordered-to-pay-250k-to-two-customers-who-lost-data-in-strasbourg-data-center-fire/ https://www.datacenterdynamics.com/en/news/ovhcloud-ordered-...
- achairapart 3y ago
  I know at least one startup that closed its doors because they lost both production and backup data on that OVH data center.
  - Snow_Falls 3y ago
    And that's why 3-2-1 specifies to have one _offsite_ backup
    - dspillett 3y ago
      Preferably with a completely different provider, in case something takes out there entire network somehow (or they go out of business). But make sure you check where the other provider is hosted: I read it one user (individual, not a startup or other company) who had backups of stuff on a shared host with a second shared host. It turned out that they were both running on servers in that one OVH DC and they were cheap hosts with no better backup/dr plan themselves...
      - pixl97 3y ago
        https://xkcd.com/908/ https://xkcd.com/908/
      - NoZebra120vClip 3y ago
        I'm gonna Craigslist all your stuff! https://xkcd.com/1150/ https://xkcd.com/1150/
- benxh 3y ago
  I personally was affected by this fire, although I've always kept 3 month backups of production data, encrypted, on-site, just in case of emergencies like this. Haven't touched their services for anything production related ever since
gary_0 3y ago
Similarly: Terrorist attack. After 9/11 I remember that some websites for WTC-based companies went offline, presumably because they were hosted on computers in the building.
- dredmorbius 3y ago
  Rather infamously, NYC's Office of Emergency Management was located in 7 WTC, and had critical communications facilities (broadcast tower) atop 1 WTC: Since 1999, New York City’s Office of Emergency Management, charged with coordinating all aspects of the response, had occupied permanent headquarters in Seven World Trade Center, on Greenwich Street, just north of the landmark twin towers. A vital communications link was the radio repeater system based on the ground floor of One World Trade Center, the north tower. The loss of those facilities – and key personnel working there – significantly hampered the response. <https://theconversation.com/disaster-communications-lessons-from-9-11-651 As you note, numerous firms and organisations experience major data losses as well, some are summarised here: <https://www.cbsnews.com/news/mystery-surrounds-loss-of-records-art-on-9-11/ https://theconversation.com/disaster-communications-lessons-...> <https://www.computerworld.com/article/2510996/9-11--top-lessons-learned-for-disaster-recovery.html https://www.computerworld.com/article/2510996/9-11--top-less...> I seem to recall an instance (I thought it was the NYC OEM, may have been another) in which both primary and secondary data archives were both within the WTC complex. Best practices now are to locate secondary / backup services at least 100--200 km from the primary. Preferably within different watersheds, seismic regions, etc. Widespread disasters are comparatively rare, but can affect considerable areas, and locations sufficiently proximate might well be affected.

by livrem 3y ago

Hamuko 3y ago

There was also the OVHcloud data centre fire in 2021, although it's debatable if it'd fall under the same "natural disaster" category. https://www.datacenterdynamics.com/en/news/ovhcloud-ordered-to-pay-250k-to-two-customers-who-lost-data-in-strasbourg-data-center-fire/ https://www.datacenterdynamics.com/en/news/ovhcloud-ordered-...

achairapart 3y ago
I know at least one startup that closed its doors because they lost both production and backup data on that OVH data center.
- Snow_Falls 3y ago
  And that's why 3-2-1 specifies to have one _offsite_ backup
  - dspillett 3y ago
    Preferably with a completely different provider, in case something takes out there entire network somehow (or they go out of business). But make sure you check where the other provider is hosted: I read it one user (individual, not a startup or other company) who had backups of stuff on a shared host with a second shared host. It turned out that they were both running on servers in that one OVH DC and they were cheap hosts with no better backup/dr plan themselves...
    - pixl97 3y ago
      https://xkcd.com/908/ https://xkcd.com/908/
    - NoZebra120vClip 3y ago
      I'm gonna Craigslist all your stuff! https://xkcd.com/1150/ https://xkcd.com/1150/
benxh 3y ago
I personally was affected by this fire, although I've always kept 3 month backups of production data, encrypted, on-site, just in case of emergencies like this. Haven't touched their services for anything production related ever since

gary_0 3y ago

Similarly: Terrorist attack. After 9/11 I remember that some websites for WTC-based companies went offline, presumably because they were hosted on computers in the building.

dredmorbius 3y ago
Rather infamously, NYC's Office of Emergency Management was located in 7 WTC, and had critical communications facilities (broadcast tower) atop 1 WTC: Since 1999, New York City’s Office of Emergency Management, charged with coordinating all aspects of the response, had occupied permanent headquarters in Seven World Trade Center, on Greenwich Street, just north of the landmark twin towers. A vital communications link was the radio repeater system based on the ground floor of One World Trade Center, the north tower. The loss of those facilities – and key personnel working there – significantly hampered the response. <https://theconversation.com/disaster-communications-lessons-from-9-11-651 As you note, numerous firms and organisations experience major data losses as well, some are summarised here: <https://www.cbsnews.com/news/mystery-surrounds-loss-of-records-art-on-9-11/ https://theconversation.com/disaster-communications-lessons-...> <https://www.computerworld.com/article/2510996/9-11--top-lessons-learned-for-disaster-recovery.html https://www.computerworld.com/article/2510996/9-11--top-less...> I seem to recall an instance (I thought it was the NYC OEM, may have been another) in which both primary and secondary data archives were both within the WTC complex. Best practices now are to locate secondary / backup services at least 100--200 km from the primary. Preferably within different watersheds, seismic regions, etc. Widespread disasters are comparatively rare, but can affect considerable areas, and locations sufficiently proximate might well be affected.