3 ms·
A colleague of mine worked for EMC in the 90s and early 2000s, and has some insane stories around 9/11. They drove him down to New York later that week, where h
by crmd 6y ago
A colleague of mine worked for EMC in the 90s and early 2000s, and has some insane stories around 9/11. They drove him down to New York later that week, where he joined a team of federal reserve, SEC, DTCC, and army combat engineers whose job was to locate at ground zero the remains of a few EMC storage arrays and pull the drives. These frames had transaction records on them that had to be recovered in order to open the markets on the 17th. The issue was that the primary and secondary replicas were in data centers in building 1 and 2. No one had ever considered that both buildings could be destroyed simultaneously. Shortly thereafter, three-site mirroring with an asynchronous leg out-of-metro became a mandatory standard for all mission critical banking data.
- nickpeterson 6y agoWow, that makes a great disaster recovery warning story
- selestify 6y agoHow successful were they? Was all of the data intact? If not, how did they deal with incomplete records for the stock market opening?
- crmd 6y agoIf I remember correctly all they needed was the last 30 seconds or so of missing data, and they were able to recover it off the drives in the symmetrix lab up in Hopkinton.
- redis_mlc 6y ago> No one had ever considered that both buildings could be destroyed simultaneously. Well, that's unimaginative. One could fall on the other since they were only 200' apart. As a DBA, I bet many people were thinking, "Look ma. 1 ms latency!" which is good, until it doesn't work.
- cafard 6y agoEMC sales types told us that the EMC method of using "vault" disks--disks that cached writes for the rest of the SAN--saved the bacon of various companies that lost power in lower Manhattan that day.