5 ms·

"Oh hey, it looks like $customer suddenly started a bunch of coinminers on their account at 10x their usual usage rate. Perfectly fine. Let them rack up a month

by olefoo 8y ago

"Oh hey, it looks like $customer suddenly started a bunch of coinminers on their account at 10x their usual usage rate. Perfectly fine. Let them rack up a months billing in a weekend; why not?" A hypothetical but not unheard of scenario in which immediate shutdown might be warranted. It's a rough world and different providers have optimised for different threat models. AWS wants to keep customers hooked; GCP wants to prevent abuse, Digital Ocean wants to show it's as capable as anyone else. If you can afford it, you build resilient multicloud infrastructure. If you can't yet do that; at the very least ensure that you have off-site backups of critical data. Cloud providers are not magic; they can fail in bizarre ways that are difficult to remedy. If you value your company you will ensure that your eggs are replicated to more than one basket and you will test your failover operations regularly. Having every deploy include failing over from one provider to another may or may not fit your comfort level; but it can be done.

lisper 8y ago
> A hypothetical but not unheard of scenario in which immediate shutdown might be warranted. Not without warning, no. It is possible that the customer intended to start a CPU-intensive process and fully intended to pay for it. Send a warning first with a specific description of the "suspicious activity" and give the customer a chance to do something about it. Don't just pull the plug with no warning.
halbritt 8y ago
"If you can afford it" There's a degree of complexity that comes with multi-cloud that's ill-suited for most early stage companies. Especially in the age of "serverless" that has folks thinking they don't need people to worry about infrastructure. My point is that the calculus has more to it than just money. The prudent response, of course, is to do as you described. Have a plan for your provider to go away. Offsite backups and the necessary config management to bring up similar infra in another region/provider is likely sufficient for most.
- jchanimal 8y ago
  It's hard to get a fully multi-cloud response in a simple stack. I describe one option for a simple multi-cloud stack in this blog post: https://blog.fauna.com/survive-cloud-vendor-crashes-with-netlify-and-faunadb https://blog.fauna.com/survive-cloud-vendor-crashes-with-net...
  - mmt 8y ago
    For an early startup, though, I would think it's not necessary to be "fully" multi-cloud. Rather, it would likely be enough to have a cloud-agnostic infrastructure with replication to a warm (or even mostly-cold to save on cost) standby at the alternate provider with a manual failover mechanism.
    - halbritt 8y ago
      Most folks overestimate their need for availability and lack a willingness to accept risk. There are distinct benefits that come with avoiding "HA" setups. Namely simplicity and speed.
      - mmt 8y ago
        > Most folks overestimate their need for availability and lack a willingness to accept risk. I disagree. More specifically, I think, instead, many [1] folks just don't make that assessment/estimate in the first place. They just follow what they perceive to be industry best practices. In many ways, this is more about social proof than a cargo cult, even though the results can resemble the latter, such as elsewhere in this thread with a comment complaining they had a "resilient" setup in a single cloud that was shut down by the provider. > There are distinct benefits that come with avoiding "HA" setups. Namely simplicity and speed. Indeed, and, perhaps more importantly, being possible at all, given time ("speed") and money ("if you can afford it"). The same could be said of "scalability" setups, which can overlap in functionality (though I would argue that in cases of overlap the dual functionality makes the cost more likely to be worth it). None of this is to say, though that "HA" is synonymous with "business continuity". It's much like the conceptual difference between RAID and backups, and even that's not always well understood. [1] I won't go so far as to say "most" because that would be a made up statistic on my part
manigandham 8y ago
> Let them rack up a months billing in a weekend Yes, there's nothing wrong with that. You have their credit card and can even authorize certain amounts ahead of time to make sure it can be charged.
- joshuamorton 8y ago
  This doesn't help if the spending is fraudulent, either because the CC is actually stolen or because it will be disputed or what have you.

by olefoo 8y ago

lisper 8y ago

> A hypothetical but not unheard of scenario in which immediate shutdown might be warranted. Not without warning, no. It is possible that the customer intended to start a CPU-intensive process and fully intended to pay for it. Send a warning first with a specific description of the "suspicious activity" and give the customer a chance to do something about it. Don't just pull the plug with no warning.

halbritt 8y ago

"If you can afford it" There's a degree of complexity that comes with multi-cloud that's ill-suited for most early stage companies. Especially in the age of "serverless" that has folks thinking they don't need people to worry about infrastructure. My point is that the calculus has more to it than just money. The prudent response, of course, is to do as you described. Have a plan for your provider to go away. Offsite backups and the necessary config management to bring up similar infra in another region/provider is likely sufficient for most.

jchanimal 8y ago
It's hard to get a fully multi-cloud response in a simple stack. I describe one option for a simple multi-cloud stack in this blog post: https://blog.fauna.com/survive-cloud-vendor-crashes-with-netlify-and-faunadb https://blog.fauna.com/survive-cloud-vendor-crashes-with-net...
- mmt 8y ago
  For an early startup, though, I would think it's not necessary to be "fully" multi-cloud. Rather, it would likely be enough to have a cloud-agnostic infrastructure with replication to a warm (or even mostly-cold to save on cost) standby at the alternate provider with a manual failover mechanism.
  - halbritt 8y ago
    Most folks overestimate their need for availability and lack a willingness to accept risk. There are distinct benefits that come with avoiding "HA" setups. Namely simplicity and speed.
    - mmt 8y ago
      > Most folks overestimate their need for availability and lack a willingness to accept risk. I disagree. More specifically, I think, instead, many [1] folks just don't make that assessment/estimate in the first place. They just follow what they perceive to be industry best practices. In many ways, this is more about social proof than a cargo cult, even though the results can resemble the latter, such as elsewhere in this thread with a comment complaining they had a "resilient" setup in a single cloud that was shut down by the provider. > There are distinct benefits that come with avoiding "HA" setups. Namely simplicity and speed. Indeed, and, perhaps more importantly, being possible at all, given time ("speed") and money ("if you can afford it"). The same could be said of "scalability" setups, which can overlap in functionality (though I would argue that in cases of overlap the dual functionality makes the cost more likely to be worth it). None of this is to say, though that "HA" is synonymous with "business continuity". It's much like the conceptual difference between RAID and backups, and even that's not always well understood. [1] I won't go so far as to say "most" because that would be a made up statistic on my part

manigandham 8y ago

> Let them rack up a months billing in a weekend Yes, there's nothing wrong with that. You have their credit card and can even authorize certain amounts ahead of time to make sure it can be charged.

joshuamorton 8y ago
This doesn't help if the spending is fraudulent, either because the CC is actually stolen or because it will be disputed or what have you.