5 ms·

There is a tale - perhaps apocryphal - handed down between generations of AWS staff, of a customer that was all-in on spot instances, until one day the price an

There is a tale - perhaps apocryphal - handed down between generations of AWS staff, of a customer that was all-in on spot instances, until one day the price and availability of their preferred instances took an unfortunate turn, which is to say, all their stuff went away, including most dramatically the customer data that was on the instance storages, and including the replicas that had been mistakenly presumed a backstop against instance loss, and sadly - but not surprisingly - this was pretty much terminal for their startup.

gonzo41 3y ago
That's like a gamma ray burst hitting the planet and just vaporizing it. So unlucky. But obviously fate had other ideas.
- sidibe 3y ago
  Not really with spot instances it was just waiting to happen on a day with more demand than usual. That is 100% expected
- sethammons 3y ago
  More like a balancing rock statue in an area that can have earthquakes
HPsquared 3y ago
Common mode failure.
BossingAround 3y ago
How..? Was there no local code on any of the dev machines? No git? I'm asking because for example if Github is vaporized today, my product would lose roughly a day or two's worth of work, since we have like 30 computers having a repository copy. Of course redeploying every single thing would not be seamless because of course, there might be some configuration stored in services, or something similar, but I'd say that ~90% of our automation is stored in Git.
- Jasp3r 3y ago
  They lost customer data, not source code. You shouldn't have a local copy of all user data on your machine.
  - appplication 3y ago
    But my customers enjoy the personal touch of me manually editing their SSN in my local MS Access database.
    - taneq 3y ago
      I mean, after all, if you don't have your own copy of the MS Access database then when your team scales beyond about 5 people that database is going to get harder to access. So really everyone should have a copy of all important PII. :P
  - pixl97 3y ago
    If something is on the cloud does 3-2-1 backup stop applying?
    - hawski 3y ago
      I would also like to know the answer. Would it be a good idea for the company to keep _encrypted_ backups on their machines/HDDs? Not a laptop somewhere, but something just a bit more involved.
      - jareklupinski 3y ago
        i think for company-critical databases, the best you can do without invoking a terrible headache for your security officer is going multi-cloud: one big tech cloud, and one smaller firm that is completely disconnected from the other one maybe they could even use a relatively inexpensive colo/baremetal provider to simply mirror the bigtech deployment on a smaller scale (would need to be quite flexible/vendor-agnostic to make that work...)
      - 3y ago
pilotneko 3y ago
Crazy, but even AWS resources are not unlimited. In 2023, I experienced multiple days where g4dn instances were not available in us-east-1 (in any AZ).
- jethro_tell 3y ago
  Heh, they still live in the real world where they have to order or build servers put them in a rack, configure them to be added to the pool. At their scale lots of their stuff is custom and needs to be ordered at least 18 months in advance. The fact that they can do capacity planning 2/3 years in advance and have very limited misses in a way that people are astonished that they have capacity misses is a testament to how good they ate at it.
  - glitchcrab 3y ago
    I think a lot of people who have never really used anything other than a major cloud provider don't really understand what goes on behind the scenes. I was a sysadmin at a hosting provider and even though we were magnitudes smaller than AWS, we still saw a lot of the same issues, just on a smaller scale.
    - jethro_tell 3y ago
      Yeah, for sure. I've got a couple team members that I brought along from cloud only, (one having never run any unmanaged k8s even) to building and maintaining on prem compute clusters. 'We had a lot of discussions along the lines of, 'Yeah, that's a great idea, but we can't just autoscale out of bad config or planning' or 'We can't just reboot a host and get a new one, we have to plan to take care of the ones we have' It's been pretty fun
darkwater 3y ago
But hey, think of the money they saved before that!
Thorrez 3y ago
The third largest bitcoin exchange made a change to their RAM settings in EC2. This shut down the machines, wiping out the hard drive and RAM. Their wallet was stored there. They lost everything. https://siliconangle.com/2011/08/01/third-largest-bitcoin-exchange-bitomat-lost-their-wallet-over-17000-bitcoins-missing/ https://siliconangle.com/2011/08/01/third-largest-bitcoin-ex...
- lnrd 3y ago
  Third largest bitcoin exchange...in 2011.
  - Synaesthesia 3y ago
    Well the bitcoin supply is limited, so it's quite a big loss.
    - stavros 3y ago
      For them! Due to the way the economy works, destroying money is actually a donation to everyone else who has money (deflation).
      - mjburgess 3y ago
        Still hard to imagine this wasnt ever just laughed out the room immediately
    - Cpoll 3y ago
      It's infinitely divisible, so not in the same sense as the loss of any other limited item. There could be only one Bitcoin in existence and it wouldn't change the utility of the system.
      - pests 3y ago
        How do you send half a sataoshi?
profsummergig 3y ago
Were they paying for said spot instances? If so, that's mostly on AWS, not the client. (Unless AWS explicitly says in its TOU that instances can be taken away instantly due to "availability" issues. Which IMHO would be a suicidal policy for AWS to have.)
- dilyevsky 3y ago
  It is not suicidal and it is in fact what it says. You get 120s notice and there’s no capacity sla even for on-demand or so called "reserved instances". You need to setup https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capacity-reservations.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capa... to get a guarantee and they will obviously bill you for it even if you're not using it.
dilyevsky 3y ago
Having all your customers' data on instances local drives (sounds like they didn't use EBS) with no backups sounds pretty dumb, spot instances or not. Those weren't serious people.

gonzo41 3y ago

That's like a gamma ray burst hitting the planet and just vaporizing it. So unlucky. But obviously fate had other ideas.

sidibe 3y ago
Not really with spot instances it was just waiting to happen on a day with more demand than usual. That is 100% expected
sethammons 3y ago
More like a balancing rock statue in an area that can have earthquakes

HPsquared 3y ago

Common mode failure.

BossingAround 3y ago

How..? Was there no local code on any of the dev machines? No git? I'm asking because for example if Github is vaporized today, my product would lose roughly a day or two's worth of work, since we have like 30 computers having a repository copy. Of course redeploying every single thing would not be seamless because of course, there might be some configuration stored in services, or something similar, but I'd say that ~90% of our automation is stored in Git.

Jasp3r 3y ago
They lost customer data, not source code. You shouldn't have a local copy of all user data on your machine.
- appplication 3y ago
  But my customers enjoy the personal touch of me manually editing their SSN in my local MS Access database.
  - taneq 3y ago
    I mean, after all, if you don't have your own copy of the MS Access database then when your team scales beyond about 5 people that database is going to get harder to access. So really everyone should have a copy of all important PII. :P
- pixl97 3y ago
  If something is on the cloud does 3-2-1 backup stop applying?
  - hawski 3y ago
    I would also like to know the answer. Would it be a good idea for the company to keep _encrypted_ backups on their machines/HDDs? Not a laptop somewhere, but something just a bit more involved.
    - jareklupinski 3y ago
      i think for company-critical databases, the best you can do without invoking a terrible headache for your security officer is going multi-cloud: one big tech cloud, and one smaller firm that is completely disconnected from the other one maybe they could even use a relatively inexpensive colo/baremetal provider to simply mirror the bigtech deployment on a smaller scale (would need to be quite flexible/vendor-agnostic to make that work...)
    - 3y ago

pilotneko 3y ago

Crazy, but even AWS resources are not unlimited. In 2023, I experienced multiple days where g4dn instances were not available in us-east-1 (in any AZ).

jethro_tell 3y ago
Heh, they still live in the real world where they have to order or build servers put them in a rack, configure them to be added to the pool. At their scale lots of their stuff is custom and needs to be ordered at least 18 months in advance. The fact that they can do capacity planning 2/3 years in advance and have very limited misses in a way that people are astonished that they have capacity misses is a testament to how good they ate at it.
- glitchcrab 3y ago
  I think a lot of people who have never really used anything other than a major cloud provider don't really understand what goes on behind the scenes. I was a sysadmin at a hosting provider and even though we were magnitudes smaller than AWS, we still saw a lot of the same issues, just on a smaller scale.
  - jethro_tell 3y ago
    Yeah, for sure. I've got a couple team members that I brought along from cloud only, (one having never run any unmanaged k8s even) to building and maintaining on prem compute clusters. 'We had a lot of discussions along the lines of, 'Yeah, that's a great idea, but we can't just autoscale out of bad config or planning' or 'We can't just reboot a host and get a new one, we have to plan to take care of the ones we have' It's been pretty fun

darkwater 3y ago

But hey, think of the money they saved before that!

Thorrez 3y ago

The third largest bitcoin exchange made a change to their RAM settings in EC2. This shut down the machines, wiping out the hard drive and RAM. Their wallet was stored there. They lost everything. https://siliconangle.com/2011/08/01/third-largest-bitcoin-exchange-bitomat-lost-their-wallet-over-17000-bitcoins-missing/ https://siliconangle.com/2011/08/01/third-largest-bitcoin-ex...

lnrd 3y ago
Third largest bitcoin exchange...in 2011.
- Synaesthesia 3y ago
  Well the bitcoin supply is limited, so it's quite a big loss.
  - stavros 3y ago
    For them! Due to the way the economy works, destroying money is actually a donation to everyone else who has money (deflation).
    - mjburgess 3y ago
      Still hard to imagine this wasnt ever just laughed out the room immediately
  - Cpoll 3y ago
    It's infinitely divisible, so not in the same sense as the loss of any other limited item. There could be only one Bitcoin in existence and it wouldn't change the utility of the system.
    - pests 3y ago
      How do you send half a sataoshi?

profsummergig 3y ago

Were they paying for said spot instances? If so, that's mostly on AWS, not the client. (Unless AWS explicitly says in its TOU that instances can be taken away instantly due to "availability" issues. Which IMHO would be a suicidal policy for AWS to have.)

dilyevsky 3y ago
It is not suicidal and it is in fact what it says. You get 120s notice and there’s no capacity sla even for on-demand or so called "reserved instances". You need to setup https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capacity-reservations.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capa... to get a guarantee and they will obviously bill you for it even if you're not using it.

dilyevsky 3y ago

Having all your customers' data on instances local drives (sounds like they didn't use EBS) with no backups sounds pretty dumb, spot instances or not. Those weren't serious people.