4 ms·

Beyond security, this highlights one of the main issues with the cloud. Was there no backup? Of course once you get beyond the headline, I think most people a

Beyond security, this highlights one of the main issues with the cloud. Was there no backup? Of course once you get beyond the headline, I think most people are much worse with protecting themselves from rare outages than Google.

mankyd 11y ago
The article states that it was "some recently written data [...] located on storage systems which were more susceptible to power failure". Perhaps it hadn't made it to a backup.
ismavis 11y ago
You find more details in the Google incident report: https://status.cloud.google.com/incident/compute/15056#5719570367119360 https://status.cloud.google.com/incident/compute/15056#57195...
bsurmanski 11y ago
As mentioned in the article, it only affected recently written data of 'Google Compute Engine' services. GCE allows user to launch VMs and generate arbitrary data on the server. Normally, Google redundantly distributes out data to at least 3 different geographically distinct locations. Check out the 'BigTable' white paper [0] for more info. For 99% of cases (and pretty well all user cases), this would not cause data lose. The key here is that the data was generated on the servers and did not have a chance to duplicate before the event. [0] http://research.google.com/archive/bigtable-osdi06.pdf http://research.google.com/archive/bigtable-osdi06.pdf
- axiak 11y ago
  Nowadays they use Reed–Solomon coding to effectively distribute their data without copying it to 3 places.
  - Artemis2 11y ago
    Do you have a source for that? Why would they start using it now?
    - magicalist 11y ago
      Also how would that help with geographic redundancy? Local recovery from errors does you no good if your datacenter gets wiped out in a flood.
    - azurezyq 11y ago
      Disclaimer: googler, but not working on storage. link here: http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.reverse-proxy.org/en/us/university/relations/facultysummit2010/storage_architecture_and_challenges.pdf http://static.googleusercontent.com/external_content/untrust... actually in colossus one can tune RS coding parameters per file, to get a tradeoff between performance/durablity. RS coding uses less copies, but same level of safety (tradeoff is the recovery computation time.)
    - axiak 11y ago
      I can't find the presentation anywhere, but at HBaseCon 2014 one of the lead developers of Bigtable stated that they went to RS. Even for their older databases. EDIT: In this video https://vimeo.com/100153741 https://vimeo.com/100153741, around the 23 minute mark
gtaylor 11y ago
> Beyond security, this highlights one of the main issues with the cloud. Was there no backup? Unless I am specifically paying for backup service, I wouldn't expect/want them to do that for me. And even if I was, I'd still have offsite backups if the system was important. If you aren't being responsible with your backups in a noisy environment like Google Cloud/AWS, understand that you are vulnerable to freak accidents like this. Google/AWS's job in all of this is to try to reduce the frequency of issues and to minimize the impact.
sowbug 11y ago
This was closer to a colocated server than a cloud service. GCE gives you a VM that you install your favorite OS on. Google keeps the VM running, but the rest (such as whether the server backs itself up) is up to you.
MichaelGG 11y ago
If you were running the data center instead of Google, it'd suddenly stop being cloud. How would that change lightning and power loss? Google Compute Engine offers customers the option to make snapshots for backup, or use a true "cloud" storage engine. If anyone lost data here it was customers explicitly not doing backups and only using a single zone. I don't know why anyone would expect different. GCE easily allows you to network machines in multiple data centers, but close geographically. So you'd only need to handle region-wide disasters.

mankyd 11y ago

The article states that it was "some recently written data [...] located on storage systems which were more susceptible to power failure". Perhaps it hadn't made it to a backup.

ismavis 11y ago

You find more details in the Google incident report: https://status.cloud.google.com/incident/compute/15056#5719570367119360 https://status.cloud.google.com/incident/compute/15056#57195...

bsurmanski 11y ago

As mentioned in the article, it only affected recently written data of 'Google Compute Engine' services. GCE allows user to launch VMs and generate arbitrary data on the server. Normally, Google redundantly distributes out data to at least 3 different geographically distinct locations. Check out the 'BigTable' white paper [0] for more info. For 99% of cases (and pretty well all user cases), this would not cause data lose. The key here is that the data was generated on the servers and did not have a chance to duplicate before the event. [0] http://research.google.com/archive/bigtable-osdi06.pdf http://research.google.com/archive/bigtable-osdi06.pdf

axiak 11y ago
Nowadays they use Reed–Solomon coding to effectively distribute their data without copying it to 3 places.
- Artemis2 11y ago
  Do you have a source for that? Why would they start using it now?
  - magicalist 11y ago
    Also how would that help with geographic redundancy? Local recovery from errors does you no good if your datacenter gets wiped out in a flood.
  - azurezyq 11y ago
    Disclaimer: googler, but not working on storage. link here: http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.reverse-proxy.org/en/us/university/relations/facultysummit2010/storage_architecture_and_challenges.pdf http://static.googleusercontent.com/external_content/untrust... actually in colossus one can tune RS coding parameters per file, to get a tradeoff between performance/durablity. RS coding uses less copies, but same level of safety (tradeoff is the recovery computation time.)
  - axiak 11y ago
    I can't find the presentation anywhere, but at HBaseCon 2014 one of the lead developers of Bigtable stated that they went to RS. Even for their older databases. EDIT: In this video https://vimeo.com/100153741 https://vimeo.com/100153741, around the 23 minute mark

gtaylor 11y ago

> Beyond security, this highlights one of the main issues with the cloud. Was there no backup? Unless I am specifically paying for backup service, I wouldn't expect/want them to do that for me. And even if I was, I'd still have offsite backups if the system was important. If you aren't being responsible with your backups in a noisy environment like Google Cloud/AWS, understand that you are vulnerable to freak accidents like this. Google/AWS's job in all of this is to try to reduce the frequency of issues and to minimize the impact.

sowbug 11y ago

This was closer to a colocated server than a cloud service. GCE gives you a VM that you install your favorite OS on. Google keeps the VM running, but the rest (such as whether the server backs itself up) is up to you.

MichaelGG 11y ago

If you were running the data center instead of Google, it'd suddenly stop being cloud. How would that change lightning and power loss? Google Compute Engine offers customers the option to make snapshots for backup, or use a true "cloud" storage engine. If anyone lost data here it was customers explicitly not doing backups and only using a single zone. I don't know why anyone would expect different. GCE easily allows you to network machines in multiple data centers, but close geographically. So you'd only need to handle region-wide disasters.