8 ms·

Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 https://twitter.com/moiseyuretsky/s

by _bxg1 7y ago

Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.

vbezhenar 7y ago
Probably because of publicity. How many of those companies went bankrupt silently, because their case did not cause much attention in news?
- _bxg1 7y ago
  Sure. Hopefully it results in a change of policy, or at least a public statement of some kind. Everyone can't depend on the cofounder to come in and save them from bad automation.
  - ummonk 7y ago
    From the looks of it they will be posting a public postmortem to their status page.
- hn_throwaway_99 7y ago
  Agreed, but it's not like the original poster had a huge platform, he just posted about it on Twitter. I may despise Twitter for a bunch of different reasons, but I can't deny it's a great tool for raising issues to companies.
  - jeremyjh 7y ago
    He has 4300 followers on twitter. What if he had 50?
- scotchio 7y ago
  Accidents can happen. Don't really blame Digital Ocean for the accidental locking but this response is insane: https://pbs.twimg.com/media/D76ocofXoAY_xB5.png https://pbs.twimg.com/media/D76ocofXoAY_xB5.png
  - mr_toad 7y ago
    > Don't really blame Digital Ocean for the accidental locking I find it concerning that they have such a low threshold for triggering a lock. 10 droplets is hardly cloud scale.
    - lbotos 7y ago
      Used to work at Linode, let's flip this on it's head: -When the majority of abuse support dealt with was people angrily calling and asking about the fraudulent charges on their cards for dozens of Lie-nodes you consider putting caps in place to reduce support burden and reduce chargebacks. At the time at Linode, if it was a known customer, we could easily and quickly raise that limit and life is good. I've always wondered how Amazon dealt with fraud/abuse at their scale. I don't think DO was wrong here to have a lock, but the post lock procedure seemed to be the problem.
  - lone_haxx0r 7y ago
    DO has shown that their service is simply not suitable for some use cases: those that impose an "unreasonable" load on their infraestructure. Even worse: they don't explicitly state what is considered "unreasonable". So, if your business is serious, you have to assume the worst-case scenario: DO can't be used for anything Conclusion: Digital Ocean is just for testing, playing around, not suitable for production.
    - deleted 7y ago
      [deleted]
    - jeremyjh 7y ago
      > Conclusion: Digital Ocean is just for testing, playing around, not suitable for production. I think that's always been the standard position most people take. DO, Linode, etc are for personal side projects, hosting community websites, forums etc. They are not for running a real business on. Some people do, sure but if hosting cost is really that big a portion of your total budget you probably don't have a real business model yet anyway.
- lazzlazzlazz 7y ago
  To be quite honest and thoughtful about it - probably extremely few companies went bankrupt silently because of issues like this. Let's be realistic.
macspoofing 7y ago
>That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places Yes they should. How many 2-man shops do you think follow all the proper backup and security procedures?
- _bxg1 7y ago
  It could literally be a cron job that dumps your DB to a desktop computer once a week. Not exactly CIA-level stuff.
  - stingraycharles 7y ago
    More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.
    - _bxg1 7y ago
      Whether it's normally a consideration or not, there are no meaningful barriers in terms of cost or effort, so it's totally realistic to expect it of a tiny startup. Every week there's another article on HN about a tiny business being squished in the gears of a giant, automated platform. In some cases like app stores this is unavoidable, but there are plenty of hosting providers to choose from. People need to learn that this is something that can happen to you in today's world, and take reasonable steps to prepare for it.
      - yourbandsucks 7y ago
        And there are a million stories of startups who build the wrong thing, don't achieve product-market fit, etc. You can't dot every I, cross every t and also build a compelling product as a 2 person shop.
        7y ago
raverbashing 7y ago
Yeah there might be some unpleasant internal conversations following that event
- whoisjuan 7y ago
  An unpleaseant conversation with his pillow. It's a two people company and he is the only technical person.
  - raverbashing 7y ago
    I'm talking about DO, not the affected company
z3t4 7y ago
Having backups in two places could easily triple the hosting costs. The question is what costs more. Eg. Losing data vs backup costs.
- h3throw 7y ago
  That seems like an odd cost increase. How do you figure it would lead to a tripling of op costs?
  - z3t4 7y ago
    Lets say you have 100 TB of data, plus two backups, you are now paying for 300 TB of data.
    - Cerium 7y ago
      I would say that it doubles the cost of backups, but using this math, we start with one copy plus one backup, and add a second backup; that means only a 50% increase.
      - Dylan16807 7y ago
        Also a secondary backup doesn't need to be in hot storage. Coldline or Glacier or similar can easily be a quarter of the price per GB.
- SteveNuts 7y ago
  As a startup, generally your secondary backup could literally be an external hard drive from best buy, or an infrequent access S3 bucket (or hell, even Glacier). No excuse, especially when "dealing with Fortune 500 companies".
- itslennysfault 7y ago
  Triple hosting costs? Literally just push a postgres dump to S3 (or any other storage provider) once a night as a "just in case something stupid happens with my primary cloud provider". It'd take a couple hours tops to set up and cost next to nothing.
taffer 7y ago
Sure, a backup would have been a significant improvement, but still – a backup only protects against data loss and not against downtime.
- fastball 7y ago
  That's probably a reason to use containerization / other technologies so that you can spin up your services in a couple minutes on a different cloud provider.
  - fgonzag 7y ago
    You don't need to use containers for that.. all you have to do is set up a warm replica of the service with another provider. The fail over doesn't even have to be automatic, but that is the minimum amount of redundancy any production SaaS should have.
    - fastball 7y ago
      A "warm replica" is going to cost money though, while containerization allows you to not have anything spun up until the moment you need it, and then have it ready to go minutes / an hour later.
      - fgonzag 7y ago
        That is patently false, unless you plan on starting from a clean slate on the new environment. Any one who proposed such a solution as a business continuity practice to me would be immediately fired. Containers solve the easy problem, which is how to make sure the dev environment matches the production environment. That is it. Replicating TBs worth of data and making sure the replica is relatively up to date is the hard part. So is fail over and fail back. Basically everything but running the code/service/app, which is the part containers solve.
        fastball 7y ago
        I was responding to this comment: > Sure, a backup would have been a significant improvement, but still – a backup only protects against data loss and not against downtime. Assuming you have data backup / recovery good to go, the downtime issue needs to be solved by getting your actual web application / logic up and running again. With something like docker-compose, you can do this on practically any provider with a couple of commands. Frontend, backend, load-balancer -- you name it, all in one command. > Containers solve the easy problem, which is how to make sure the dev environment matches the production environment. That is it. Speaking of "patently false"...
jrochkind1 7y ago
Uretsky: > Account should be re-activated - need to look deeper into the way this was handled. It shouldn't have taken this long to get the account back up, and also for it not be flagged a second time. So... he doesn't address what is the scariest part to me, the message that just says "Nope, we've decided never to give your account back, it's gone, the end."
- dahfizz 7y ago
  How would you like that to have been addressed? I think it's entirely reasonable for companies to have that option. "You are doing something malicious and against the rules, you have been permanently removed". In this case, that option was misused, but I don't think the existence of that possiblity is inheritly surprising.
  - jwr 7y ago
    Access to your data should never be denied. Ever. It was not DigitalOcean's data. If you are a hosting provider, you can't ever hold customer data hostage or deny them access to it in any way.
    - dahfizz 7y ago
      Again, I must disagree. If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it. DO should not be forced to aid bad actors. And, regardless of what DO should or should not do, they can do whatever they want with their own hard drives. You should structure your business accordingly.
      - duskwuff 7y ago
        For some practical, if extreme, examples: if a customer were to host a phishing site, or a site hosting CP, it would be grossly irresponsible (and likely even illegal) for the hosting provider to retain the customer's data after account suspension and allow them to download it.
DKnoll 7y ago
>That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. They should have, at the very least, one DR site on a different provider in a different region that is replicated in real-time and ready to go live after an outage is confirmed by the IT Operations team (or automatically depending on what services are being run).

by _bxg1 7y ago

vbezhenar 7y ago

Probably because of publicity. How many of those companies went bankrupt silently, because their case did not cause much attention in news?

_bxg1 7y ago
Sure. Hopefully it results in a change of policy, or at least a public statement of some kind. Everyone can't depend on the cofounder to come in and save them from bad automation.
- ummonk 7y ago
  From the looks of it they will be posting a public postmortem to their status page.
hn_throwaway_99 7y ago
Agreed, but it's not like the original poster had a huge platform, he just posted about it on Twitter. I may despise Twitter for a bunch of different reasons, but I can't deny it's a great tool for raising issues to companies.
- jeremyjh 7y ago
  He has 4300 followers on twitter. What if he had 50?
scotchio 7y ago
Accidents can happen. Don't really blame Digital Ocean for the accidental locking but this response is insane: https://pbs.twimg.com/media/D76ocofXoAY_xB5.png https://pbs.twimg.com/media/D76ocofXoAY_xB5.png
- mr_toad 7y ago
  > Don't really blame Digital Ocean for the accidental locking I find it concerning that they have such a low threshold for triggering a lock. 10 droplets is hardly cloud scale.
  - lbotos 7y ago
    Used to work at Linode, let's flip this on it's head: -When the majority of abuse support dealt with was people angrily calling and asking about the fraudulent charges on their cards for dozens of Lie-nodes you consider putting caps in place to reduce support burden and reduce chargebacks. At the time at Linode, if it was a known customer, we could easily and quickly raise that limit and life is good. I've always wondered how Amazon dealt with fraud/abuse at their scale. I don't think DO was wrong here to have a lock, but the post lock procedure seemed to be the problem.
- lone_haxx0r 7y ago
  DO has shown that their service is simply not suitable for some use cases: those that impose an "unreasonable" load on their infraestructure. Even worse: they don't explicitly state what is considered "unreasonable". So, if your business is serious, you have to assume the worst-case scenario: DO can't be used for anything Conclusion: Digital Ocean is just for testing, playing around, not suitable for production.
  - deleted 7y ago
    [deleted]
  - jeremyjh 7y ago
    > Conclusion: Digital Ocean is just for testing, playing around, not suitable for production. I think that's always been the standard position most people take. DO, Linode, etc are for personal side projects, hosting community websites, forums etc. They are not for running a real business on. Some people do, sure but if hosting cost is really that big a portion of your total budget you probably don't have a real business model yet anyway.
lazzlazzlazz 7y ago
To be quite honest and thoughtful about it - probably extremely few companies went bankrupt silently because of issues like this. Let's be realistic.

macspoofing 7y ago

>That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places Yes they should. How many 2-man shops do you think follow all the proper backup and security procedures?

_bxg1 7y ago
It could literally be a cron job that dumps your DB to a desktop computer once a week. Not exactly CIA-level stuff.
- stingraycharles 7y ago
  More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.
  - _bxg1 7y ago
    Whether it's normally a consideration or not, there are no meaningful barriers in terms of cost or effort, so it's totally realistic to expect it of a tiny startup. Every week there's another article on HN about a tiny business being squished in the gears of a giant, automated platform. In some cases like app stores this is unavoidable, but there are plenty of hosting providers to choose from. People need to learn that this is something that can happen to you in today's world, and take reasonable steps to prepare for it.
    - yourbandsucks 7y ago
      And there are a million stories of startups who build the wrong thing, don't achieve product-market fit, etc. You can't dot every I, cross every t and also build a compelling product as a 2 person shop.
      - 7y ago

raverbashing 7y ago

Yeah there might be some unpleasant internal conversations following that event

whoisjuan 7y ago
An unpleaseant conversation with his pillow. It's a two people company and he is the only technical person.
- raverbashing 7y ago
  I'm talking about DO, not the affected company

z3t4 7y ago

Having backups in two places could easily triple the hosting costs. The question is what costs more. Eg. Losing data vs backup costs.

h3throw 7y ago
That seems like an odd cost increase. How do you figure it would lead to a tripling of op costs?
- z3t4 7y ago
  Lets say you have 100 TB of data, plus two backups, you are now paying for 300 TB of data.
  - Cerium 7y ago
    I would say that it doubles the cost of backups, but using this math, we start with one copy plus one backup, and add a second backup; that means only a 50% increase.
    - Dylan16807 7y ago
      Also a secondary backup doesn't need to be in hot storage. Coldline or Glacier or similar can easily be a quarter of the price per GB.
SteveNuts 7y ago
As a startup, generally your secondary backup could literally be an external hard drive from best buy, or an infrequent access S3 bucket (or hell, even Glacier). No excuse, especially when "dealing with Fortune 500 companies".
itslennysfault 7y ago
Triple hosting costs? Literally just push a postgres dump to S3 (or any other storage provider) once a night as a "just in case something stupid happens with my primary cloud provider". It'd take a couple hours tops to set up and cost next to nothing.

taffer 7y ago

Sure, a backup would have been a significant improvement, but still – a backup only protects against data loss and not against downtime.

fastball 7y ago
That's probably a reason to use containerization / other technologies so that you can spin up your services in a couple minutes on a different cloud provider.
- fgonzag 7y ago
  You don't need to use containers for that.. all you have to do is set up a warm replica of the service with another provider. The fail over doesn't even have to be automatic, but that is the minimum amount of redundancy any production SaaS should have.
  - fastball 7y ago
    A "warm replica" is going to cost money though, while containerization allows you to not have anything spun up until the moment you need it, and then have it ready to go minutes / an hour later.
    - fgonzag 7y ago
      That is patently false, unless you plan on starting from a clean slate on the new environment. Any one who proposed such a solution as a business continuity practice to me would be immediately fired. Containers solve the easy problem, which is how to make sure the dev environment matches the production environment. That is it. Replicating TBs worth of data and making sure the replica is relatively up to date is the hard part. So is fail over and fail back. Basically everything but running the code/service/app, which is the part containers solve.
      - fastball 7y ago
        I was responding to this comment: > Sure, a backup would have been a significant improvement, but still – a backup only protects against data loss and not against downtime. Assuming you have data backup / recovery good to go, the downtime issue needs to be solved by getting your actual web application / logic up and running again. With something like docker-compose, you can do this on practically any provider with a couple of commands. Frontend, backend, load-balancer -- you name it, all in one command. > Containers solve the easy problem, which is how to make sure the dev environment matches the production environment. That is it. Speaking of "patently false"...

jrochkind1 7y ago

Uretsky: > Account should be re-activated - need to look deeper into the way this was handled. It shouldn't have taken this long to get the account back up, and also for it not be flagged a second time. So... he doesn't address what is the scariest part to me, the message that just says "Nope, we've decided never to give your account back, it's gone, the end."

dahfizz 7y ago
How would you like that to have been addressed? I think it's entirely reasonable for companies to have that option. "You are doing something malicious and against the rules, you have been permanently removed". In this case, that option was misused, but I don't think the existence of that possiblity is inheritly surprising.
- jwr 7y ago
  Access to your data should never be denied. Ever. It was not DigitalOcean's data. If you are a hosting provider, you can't ever hold customer data hostage or deny them access to it in any way.
  - dahfizz 7y ago
    Again, I must disagree. If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it. DO should not be forced to aid bad actors. And, regardless of what DO should or should not do, they can do whatever they want with their own hard drives. You should structure your business accordingly.
    - duskwuff 7y ago
      For some practical, if extreme, examples: if a customer were to host a phishing site, or a site hosting CP, it would be grossly irresponsible (and likely even illegal) for the hosting provider to retain the customer's data after account suspension and allow them to download it.

DKnoll 7y ago

>That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. They should have, at the very least, one DR site on a different provider in a different region that is replicated in real-time and ready to go live after an outage is confirmed by the IT Operations team (or automatically depending on what services are being run).