14 ms·

Could you provide more details?

sneak 3y ago
On delete of VMs, the next user to be allocated disk would be able to read your data. There was a "scrub" feature that would bill you longer/extra to zeroize your data and not give it to the next customer. Apparently they hadn't heard of TRIM or thin provisioning. https://github.com/fog/fog/issues/2525 https://github.com/fog/fog/issues/2525 https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983520 https://news.ycombinator.com/item?id=6983520
- belter 3y ago
  Some of the comments by DO on that GitHub thread, are really surprising, and disappointing to say the least...
  - neom 3y ago
    I wrote those comments, fog wasn't using the API as he would have liked, and then this guy starts a bunch of trouble on HN stating that there was a vun on DigitalOcean, and conflated someone using the API incorrectly with us having a security vun. Two things I was trying to (admittedly not very well) say was: if DO did have a security issue, posting it on HN ("DigitalOcean leaks customer data between VMs" huh?) instead of emailing us about it isn't a very responsible way to disclose it and, there isn't actually a security issue, the fog app wasn't passing the scrub flag. In retrospect, I should have been less of a jerk, but I still think sneak was acting in bad faith in how he went about the whole thing. 10 years later and I still remember how pissed off I was that day, hah.
    - richbell 3y ago
      I do find the claim that "there's nothing irresponsible about full immediate disclosure" to be interesting. https://github.com/fog/fog/issues/2525#issuecomment-31336855 https://github.com/fog/fog/issues/2525#issuecomment-31336855
      - deleted 3y ago
        [deleted]
    - ratg13 3y ago
      The fact that you think it should be the customers’ responsibility to make sure that DO doesn’t accidentally give your private data away to unknown third parties is an absolutely minblowing take on this matter. I’ve been a DO customer for the better part of a decade, but there is no way that I can continue to be a customer with a company that has such a blasé stance toward security and protecting customer data. If anyone can suggest any alternatives that are not Hetzner, I would be interested.
      - neom 3y ago
        Well first, I've not worked for DigitalOcean in a long time, I left in 2015. I don't really wanna re-hash something from 2013, but again: when you deleted a VM you could click the scrub button or pass the scrub flag in the API when you issued a delete call. It was very well documented and the scrub button wasn't hidden anywhere in the interface. The fog platform wasn't using the scrub flag in their app for whatever reason. As I recall, we did change it to scrub by default shortly after. From what I remember, the reason it wasn't originally default was the zero'ing process tried up a lot of resource on the box and it was hard on the drives and we were very much in startup mode at that time with limited capacity, so it was better to only zero if the customer requests it.
        mdpye 3y ago
        You still don't appear to get it though. The fact that there even exists an option (never mind a default) which does not scrub storage between use by different tenants is completely unacceptable in a multi-tenant environment.
        ratg13 3y ago
        Imagine if your bank told you they only close the vault when a customer asks. And they give you an excuse that it’s time consuming for them and you should respect their time. Would you bank with them? This is a pathetic response and you are dragging your former company through the mud by putting the incompetence that you participated in, front and center. It’s amazing that this happened so long ago and yet you seemed to have learned nothing from it, and now want to double down on your right to leak sensitive data (because it was convenient at the time)
    - sneak 3y ago
      Wow, this just confirms my belief. There is precisely zero circumstance where it is ok to give private customer data to another customer. The fact that you think it had anything to do with an API or how it is used is all anyone needs to know about it. The idea that warning your customers of your vulnerabilities is "irresponsible" is only true if you care more about revenues than your customers' security.
      - plagiarist 3y ago
        Unless I misunderstood something and one has to explicitly opt out of scrub for this to occur, I agree 100%. This is surprising. And they were mad that a customer thinks this is undesirable? I was using DO. It is not a serious project but I think I'll be moving on nonetheless.
        Operyl 3y ago
        For what it's worth, neom has moved on from DO a while back.
        plagiarist 3y ago
        Yeah, I read a bit further and it was not scrub by default but had to be opt-in. And today is the first I heard that all API keys have ridiculous access. I'm done. Now I just need to find a cloud hosting service that will do a low cost easy k8s cluster, preferably not AWS.
        Operyl 3y ago
        Looks like OVH does managed k8s, and they're pretty solid at this point for the most part. https://us.ovhcloud.com/public-cloud/kubernetes/ https://us.ovhcloud.com/public-cloud/kubernetes/
        plagiarist 3y ago
        Thank you, I didn't know they did. That puts them top-of-list for me.
    - plagiarist 3y ago
      Why on Earth would someone have to opt-in to scrubbing with a flag vs. opting out? That's surprising. Is the flag on by default and they specifically opted out before complaining about it?
      - fullspectrumdev 3y ago
        The scrub flag was default false, apparently because scrubbing was slow and caused wear.
    - rsynnott 3y ago
      > there isn't actually a security issue, the fog app wasn't passing the scrub flag At a certain point a default behaviour can be so bad, and so clearly not what a user would expect or want, that it constitutes a security issue. I would think this _more_ than qualifies.
      - neom 3y ago
        You're right. I hadn't really thought about it that way. My response at the time actually came from confusion on my part (that is my own fault I suppose). I got woken up at 4am on a Sunday by a flood of calls saying there is a post on the top of HN about DO leaking data between VMs. I got out of bed to see if I needed to wake up the engineering team, checked the support system and the disclosure emails because my first thought was shit someone has pwned the LXC/escaped the jails or something. Then when I read the github, I thought the scrub flag wasn't being respected when it was passed. That was the type of security issue I had originally taken it to mean, both I would have woken up engineering for. It was a someone is in the hypervisor type of security issue I thought we were dealing with. Once I realized it was fog didn't pass the scrub flag, I wish I'd just gone back to bed and dealt with it on Monday instead of trying to do it in real time, that was stupid of me. I also accept your point that an unexpected behaviour can in itself be a security issue.
    - MarkSweep 3y ago
      > there isn't actually a security issue The official blog from DigitalOcean is here: https://www.digitalocean.com/blog/transparency-regarding-data-security https://www.digitalocean.com/blog/transparency-regarding-dat... While it does not call it a “security issue” directly, it details several changes to the service to prevent customer data leaking between accounts. That seems to contradict your position that there was no problem. > 10 years later and I still remember how pissed off I was that day, hah. I would have hoped that after 10 years you would be able to admit that letting customers read each other’s data was a mistake. The existence of a disk scrub API, the problem being improperly reported, and DO being advertised as “not for production use” are not valid excuses.
    - figassis 3y ago
      Consider the multiple S3 leaks that caused AWS to change their defaults to block public access and disable ACLs. You could say that since buckets were always default private, any misconfiguration was simply user error and users complaining were doing so in bad faith. Or you could understand the implications and help your customers protect their data.
- 15155 3y ago
  I actually came here to add this very issue and then noticed this comment. This issue is the reason why I will never use DO or recommend it to a customer.
- fullspectrumdev 3y ago
  Holy shit that’s bad. I wonder how one could test for that “automatically” on other providers.

On delete of VMs, the next user to be allocated disk would be able to read your data. There was a "scrub" feature that would bill you longer/extra to zeroize your data and not give it to the next customer. Apparently they hadn't heard of TRIM or thin provisioning. https://github.com/fog/fog/issues/2525 https://github.com/fog/fog/issues/2525 https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983520 https://news.ycombinator.com/item?id=6983520

belter 3y ago
Some of the comments by DO on that GitHub thread, are really surprising, and disappointing to say the least...
- neom 3y ago
  I wrote those comments, fog wasn't using the API as he would have liked, and then this guy starts a bunch of trouble on HN stating that there was a vun on DigitalOcean, and conflated someone using the API incorrectly with us having a security vun. Two things I was trying to (admittedly not very well) say was: if DO did have a security issue, posting it on HN ("DigitalOcean leaks customer data between VMs" huh?) instead of emailing us about it isn't a very responsible way to disclose it and, there isn't actually a security issue, the fog app wasn't passing the scrub flag. In retrospect, I should have been less of a jerk, but I still think sneak was acting in bad faith in how he went about the whole thing. 10 years later and I still remember how pissed off I was that day, hah.
  - richbell 3y ago
    I do find the claim that "there's nothing irresponsible about full immediate disclosure" to be interesting. https://github.com/fog/fog/issues/2525#issuecomment-31336855 https://github.com/fog/fog/issues/2525#issuecomment-31336855
    - deleted 3y ago
      [deleted]
  - ratg13 3y ago
    The fact that you think it should be the customers’ responsibility to make sure that DO doesn’t accidentally give your private data away to unknown third parties is an absolutely minblowing take on this matter. I’ve been a DO customer for the better part of a decade, but there is no way that I can continue to be a customer with a company that has such a blasé stance toward security and protecting customer data. If anyone can suggest any alternatives that are not Hetzner, I would be interested.
    - neom 3y ago
      Well first, I've not worked for DigitalOcean in a long time, I left in 2015. I don't really wanna re-hash something from 2013, but again: when you deleted a VM you could click the scrub button or pass the scrub flag in the API when you issued a delete call. It was very well documented and the scrub button wasn't hidden anywhere in the interface. The fog platform wasn't using the scrub flag in their app for whatever reason. As I recall, we did change it to scrub by default shortly after. From what I remember, the reason it wasn't originally default was the zero'ing process tried up a lot of resource on the box and it was hard on the drives and we were very much in startup mode at that time with limited capacity, so it was better to only zero if the customer requests it.
      - mdpye 3y ago
        You still don't appear to get it though. The fact that there even exists an option (never mind a default) which does not scrub storage between use by different tenants is completely unacceptable in a multi-tenant environment.
      - ratg13 3y ago
        Imagine if your bank told you they only close the vault when a customer asks. And they give you an excuse that it’s time consuming for them and you should respect their time. Would you bank with them? This is a pathetic response and you are dragging your former company through the mud by putting the incompetence that you participated in, front and center. It’s amazing that this happened so long ago and yet you seemed to have learned nothing from it, and now want to double down on your right to leak sensitive data (because it was convenient at the time)
  - sneak 3y ago
    Wow, this just confirms my belief. There is precisely zero circumstance where it is ok to give private customer data to another customer. The fact that you think it had anything to do with an API or how it is used is all anyone needs to know about it. The idea that warning your customers of your vulnerabilities is "irresponsible" is only true if you care more about revenues than your customers' security.
    - plagiarist 3y ago
      Unless I misunderstood something and one has to explicitly opt out of scrub for this to occur, I agree 100%. This is surprising. And they were mad that a customer thinks this is undesirable? I was using DO. It is not a serious project but I think I'll be moving on nonetheless.
      - Operyl 3y ago
        For what it's worth, neom has moved on from DO a while back.
        plagiarist 3y ago
        Yeah, I read a bit further and it was not scrub by default but had to be opt-in. And today is the first I heard that all API keys have ridiculous access. I'm done. Now I just need to find a cloud hosting service that will do a low cost easy k8s cluster, preferably not AWS.
        Operyl 3y ago
        Looks like OVH does managed k8s, and they're pretty solid at this point for the most part. https://us.ovhcloud.com/public-cloud/kubernetes/ https://us.ovhcloud.com/public-cloud/kubernetes/
        plagiarist 3y ago
        Thank you, I didn't know they did. That puts them top-of-list for me.
  - plagiarist 3y ago
    Why on Earth would someone have to opt-in to scrubbing with a flag vs. opting out? That's surprising. Is the flag on by default and they specifically opted out before complaining about it?
    - fullspectrumdev 3y ago
      The scrub flag was default false, apparently because scrubbing was slow and caused wear.
  - rsynnott 3y ago
    > there isn't actually a security issue, the fog app wasn't passing the scrub flag At a certain point a default behaviour can be so bad, and so clearly not what a user would expect or want, that it constitutes a security issue. I would think this _more_ than qualifies.
    - neom 3y ago
      You're right. I hadn't really thought about it that way. My response at the time actually came from confusion on my part (that is my own fault I suppose). I got woken up at 4am on a Sunday by a flood of calls saying there is a post on the top of HN about DO leaking data between VMs. I got out of bed to see if I needed to wake up the engineering team, checked the support system and the disclosure emails because my first thought was shit someone has pwned the LXC/escaped the jails or something. Then when I read the github, I thought the scrub flag wasn't being respected when it was passed. That was the type of security issue I had originally taken it to mean, both I would have woken up engineering for. It was a someone is in the hypervisor type of security issue I thought we were dealing with. Once I realized it was fog didn't pass the scrub flag, I wish I'd just gone back to bed and dealt with it on Monday instead of trying to do it in real time, that was stupid of me. I also accept your point that an unexpected behaviour can in itself be a security issue.
  - MarkSweep 3y ago
    > there isn't actually a security issue The official blog from DigitalOcean is here: https://www.digitalocean.com/blog/transparency-regarding-data-security https://www.digitalocean.com/blog/transparency-regarding-dat... While it does not call it a “security issue” directly, it details several changes to the service to prevent customer data leaking between accounts. That seems to contradict your position that there was no problem. > 10 years later and I still remember how pissed off I was that day, hah. I would have hoped that after 10 years you would be able to admit that letting customers read each other’s data was a mistake. The existence of a disk scrub API, the problem being improperly reported, and DO being advertised as “not for production use” are not valid excuses.
  - figassis 3y ago
    Consider the multiple S3 leaks that caused AWS to change their defaults to block public access and disable ACLs. You could say that since buckets were always default private, any misconfiguration was simply user error and users complaining were doing so in bad faith. Or you could understand the implications and help your customers protect their data.
15155 3y ago
I actually came here to add this very issue and then noticed this comment. This issue is the reason why I will never use DO or recommend it to a customer.
fullspectrumdev 3y ago
Holy shit that’s bad. I wonder how one could test for that “automatically” on other providers.