4 ms·

Transparency Regarding Data Security

53 points by chrishas35 13y ago

sneak 13y ago
> At no time was customer data "leaked" between accounts. This would require that a user explicitly not scrub their volume after destroying their server; in this instance data would be recoverable and should be considered not sensitive. For fuck's sake, now you're just lying. Not scrubbing has been the default - a user doesn't have to "explicitly not scrub". A simple unadorned "destroy" api command creates the circumstance in which the data is not destroyed, but made available to others. If no customer data leaked between accounts, how was I able to read someone else's stack traces[1], web logs[2], and customer tokens[3] on a freshly provisioned VM? What follows is evidence to directly support the claim that you're lying through your teeth to save face after having been caught being grossly irresponsible with your customers' data. Please start acting like professionals. [1] http://i.imgur.com/TMp2kdf.png http://i.imgur.com/TMp2kdf.png [2] http://i.imgur.com/WLv2qSE.png http://i.imgur.com/WLv2qSE.png [3] http://i.imgur.com/fJOxRN9.png http://i.imgur.com/fJOxRN9.png
- nodesocket 13y ago
  You're being ridiculous. They made a conscious decision to increase performance and the lifetime of their SSD drives by turning `scrub` off by default. If you enjoy DigitalOcean's prices (starting at $5), there must be some balance and optimizations to continue to run a business. In the control panel, its quite clear, and easy to simply check scrub on. I agree that this change on the API front should have been articulated. However, Moisey acknowledge their mistake, let's move past it.
  - thirsteh 13y ago
    No. It's not ridiculous. What's ridiculous is that they're even trying to claim no data was leaked when it demonstrably was. There are many ways to securely scrub an SSD without subjecting it to a full write. They were outlined in the original thread.
  - integraton 13y ago
    Customers should not be able to access other customers' data under any circumstances.
  - sneak 13y ago
    This has nothing to do with scrubbing, or SSDs. This has to do with them providing my data to third parties without my consent. They've leaked customer data between VMs. Their claims that this isn't true are false. Look at the screenshots!
  - deleted 13y ago
    [deleted]
  - corresation 13y ago
    No one would intentionally make the choice of having their data shared with other users, and if that is a business model they rely upon they need to find another. This is a profound, never-ever-should-happen gap in data security, and it is yet another instance where DigitalOcean comes out looking remarkably amateurish. Conversationally, I would have expected their hypervisor to have been doing thin provisioning: That until you write data to a block the block is 0s (having no provisioning on actual storage). And if you write 0s, that too isn't actually provisioned on real storage. And when you write actual data, well that is what the block now contains.
    - gizmo686 13y ago
      >No one would intentionally make the choice of having their data shared with other users, and if that is a business model they rely upon they need to find another. Unless they do not care if their data is shared, and would be willing to let it be shared for a reduced cost.
      - corresation 13y ago
        If there were a box that said "share your data" that you had to click, I cannot imagine the users who would click it. Further no one was ever told that the value proposition of DigitalOcean relies upon a complete and utter lack of data integrity. That is a dishonest angle. It is not in any way how this has been sold. Further you don't get a discount for choosing not to scrub, but instead essentially get tricked into it by the magical law of defaults.
        gizmo686 13y ago
        I do not mean to say that it was justified in this particular case. I was responding to the idea that all products must be secure, even for users who do not want to pay for the added security. Also, it is not so much that I think a service should offer a discount for handling your data insecurely (although in this particular case, the insecure handling is actually cheaper on a per user basis). Rather, there is a place in the market for products that are not secure, and therefore do not have to pay to develop and maintain the security aspects of the service.
  - bradleyland 13y ago
    And yet all the other major VPS/cloud server providers have engineered solutions that do not suffer from this issue (regardless of physical storage medium). Specifically, Amazon, Linode, and Rackspace all have automatic mitigations (in varying forms) that prevent this type of leak.
- nemesisj 13y ago
  Agreed. This is a lie. Plain and simple. Even of you don't know any of the particulars about this issue, the post just contradicts itself. This is a textbook example of how not to own up to a security issue - if anything, this response makes the entire thing worse because you realise the actual problem (their approach) isn't being fixed, only this specific manifestation of the issue.
- whyme 13y ago
  I read that they acknowledge they made the default to not scrub... "As a result, we switched the default mode away from scrubbing to improve performance, given that customers would have complete control over this action themselves." Also, I think they presume a leak to be qualified by the data being sensitive which would not be leaked had it been scrubbed, for which they provide an option. This has to be assumed, otherwise one would never need to scrub. Still, certainly looks to be a pretty big mistake in communications.
- kordless 13y ago
  > you're just lying You are aware you are talking to more than one person at DigitalOcean, right? Their post has contradictions in it because the person that wrote it isn't the same person who is implementing the fix. Clearly they know they fucked up, are fixing the fuck up, and are trying to implement damage control. The people doing the fix for the fuckup are fixing things because they agree with you. The people writing the copy for the damage control don't agree with you. It's a clear case of cognitive dissonance, which isn't surprising given there is more than one person involved over there. Individuals already have cognitive dissonance to begin with, it just gets worse when more people are involved. A company without cognitive dissonance would have never implemented this to begin with. Celebrate the fact you got them to change it and move on. Fighting them with blaming statements is just going to make the people who actually fixed it for you sad and disappointed. That's not what any of us want. And, you are right, someone over there is just lying and someone else over there in charge should take them aside and explain how it's not cool.
  - bradleyland 13y ago
    Engineers make this mistake all the time, but it's important to remember that for everyone outside our field, the fact that you can explain why something is happening does not make it ok. Things that are wrong with this communication: * There are contradictions in claim * There are misstatements of fact * They are offering a poorly qualified apology > Fighting them with blaming statements is just going to make the people who actually fixed it for you sad and disappointed. Disagree 100%. If the management team can't get their head around the "right" decisions from an engineering perspective, the engineers are in for a world of sadness and disappointment. The management at DO need to hear every one of the criticisms being leveled here and on their own communications channels.
  - nknighthb 13y ago
    I will agree not to blame corporations for the actions of their employees the moment those employees are individually liable for their actions. Corporations always speak with one voice. If that voice says something wrong or inconsistent, it is not the public's problem, it is the corporation's. No excuses. Ever.
alimoeeny 13y ago
I am not their customer, but I admire their acceptance of their mistake, "The second mistake that we made was not notifying our customers that use the API"
ak217 13y ago
The fact that DigitalOcean have no idea how to secure customer block devices makes me wonder what other security issues they are unaware of.
callesgg 13y ago
Either digital ocean actually has real SSd disks for each vm or something is deeply wrong. A new VM should have a new disk file. When I create a vm I create a vmdk file that represents the disk. When I delete the vm I also delete the vmdk file. If I then create a new vm I would use a new vmdk file. Apparently that is not the way digital ocean does things, but how do they do stuff then?
- wmf 13y ago
  This was covered in the previous thread: https://news.ycombinator.com/item?id=6983270 https://news.ycombinator.com/item?id=6983270 DO may not be using file-based storage because it is slower.

53 points by chrishas35 13y ago

sneak 13y ago

> At no time was customer data "leaked" between accounts. This would require that a user explicitly not scrub their volume after destroying their server; in this instance data would be recoverable and should be considered not sensitive. For fuck's sake, now you're just lying. Not scrubbing has been the default - a user doesn't have to "explicitly not scrub". A simple unadorned "destroy" api command creates the circumstance in which the data is not destroyed, but made available to others. If no customer data leaked between accounts, how was I able to read someone else's stack traces[1], web logs[2], and customer tokens[3] on a freshly provisioned VM? What follows is evidence to directly support the claim that you're lying through your teeth to save face after having been caught being grossly irresponsible with your customers' data. Please start acting like professionals. [1] http://i.imgur.com/TMp2kdf.png http://i.imgur.com/TMp2kdf.png [2] http://i.imgur.com/WLv2qSE.png http://i.imgur.com/WLv2qSE.png [3] http://i.imgur.com/fJOxRN9.png http://i.imgur.com/fJOxRN9.png

nodesocket 13y ago
You're being ridiculous. They made a conscious decision to increase performance and the lifetime of their SSD drives by turning `scrub` off by default. If you enjoy DigitalOcean's prices (starting at $5), there must be some balance and optimizations to continue to run a business. In the control panel, its quite clear, and easy to simply check scrub on. I agree that this change on the API front should have been articulated. However, Moisey acknowledge their mistake, let's move past it.
- thirsteh 13y ago
  No. It's not ridiculous. What's ridiculous is that they're even trying to claim no data was leaked when it demonstrably was. There are many ways to securely scrub an SSD without subjecting it to a full write. They were outlined in the original thread.
- integraton 13y ago
  Customers should not be able to access other customers' data under any circumstances.
- sneak 13y ago
  This has nothing to do with scrubbing, or SSDs. This has to do with them providing my data to third parties without my consent. They've leaked customer data between VMs. Their claims that this isn't true are false. Look at the screenshots!
- deleted 13y ago
  [deleted]
- corresation 13y ago
  No one would intentionally make the choice of having their data shared with other users, and if that is a business model they rely upon they need to find another. This is a profound, never-ever-should-happen gap in data security, and it is yet another instance where DigitalOcean comes out looking remarkably amateurish. Conversationally, I would have expected their hypervisor to have been doing thin provisioning: That until you write data to a block the block is 0s (having no provisioning on actual storage). And if you write 0s, that too isn't actually provisioned on real storage. And when you write actual data, well that is what the block now contains.
  - gizmo686 13y ago
    >No one would intentionally make the choice of having their data shared with other users, and if that is a business model they rely upon they need to find another. Unless they do not care if their data is shared, and would be willing to let it be shared for a reduced cost.
    - corresation 13y ago
      If there were a box that said "share your data" that you had to click, I cannot imagine the users who would click it. Further no one was ever told that the value proposition of DigitalOcean relies upon a complete and utter lack of data integrity. That is a dishonest angle. It is not in any way how this has been sold. Further you don't get a discount for choosing not to scrub, but instead essentially get tricked into it by the magical law of defaults.
      - gizmo686 13y ago
        I do not mean to say that it was justified in this particular case. I was responding to the idea that all products must be secure, even for users who do not want to pay for the added security. Also, it is not so much that I think a service should offer a discount for handling your data insecurely (although in this particular case, the insecure handling is actually cheaper on a per user basis). Rather, there is a place in the market for products that are not secure, and therefore do not have to pay to develop and maintain the security aspects of the service.
- bradleyland 13y ago
  And yet all the other major VPS/cloud server providers have engineered solutions that do not suffer from this issue (regardless of physical storage medium). Specifically, Amazon, Linode, and Rackspace all have automatic mitigations (in varying forms) that prevent this type of leak.
nemesisj 13y ago
Agreed. This is a lie. Plain and simple. Even of you don't know any of the particulars about this issue, the post just contradicts itself. This is a textbook example of how not to own up to a security issue - if anything, this response makes the entire thing worse because you realise the actual problem (their approach) isn't being fixed, only this specific manifestation of the issue.
whyme 13y ago
I read that they acknowledge they made the default to not scrub... "As a result, we switched the default mode away from scrubbing to improve performance, given that customers would have complete control over this action themselves." Also, I think they presume a leak to be qualified by the data being sensitive which would not be leaked had it been scrubbed, for which they provide an option. This has to be assumed, otherwise one would never need to scrub. Still, certainly looks to be a pretty big mistake in communications.
kordless 13y ago
> you're just lying You are aware you are talking to more than one person at DigitalOcean, right? Their post has contradictions in it because the person that wrote it isn't the same person who is implementing the fix. Clearly they know they fucked up, are fixing the fuck up, and are trying to implement damage control. The people doing the fix for the fuckup are fixing things because they agree with you. The people writing the copy for the damage control don't agree with you. It's a clear case of cognitive dissonance, which isn't surprising given there is more than one person involved over there. Individuals already have cognitive dissonance to begin with, it just gets worse when more people are involved. A company without cognitive dissonance would have never implemented this to begin with. Celebrate the fact you got them to change it and move on. Fighting them with blaming statements is just going to make the people who actually fixed it for you sad and disappointed. That's not what any of us want. And, you are right, someone over there is just lying and someone else over there in charge should take them aside and explain how it's not cool.
- bradleyland 13y ago
  Engineers make this mistake all the time, but it's important to remember that for everyone outside our field, the fact that you can explain why something is happening does not make it ok. Things that are wrong with this communication: * There are contradictions in claim * There are misstatements of fact * They are offering a poorly qualified apology > Fighting them with blaming statements is just going to make the people who actually fixed it for you sad and disappointed. Disagree 100%. If the management team can't get their head around the "right" decisions from an engineering perspective, the engineers are in for a world of sadness and disappointment. The management at DO need to hear every one of the criticisms being leveled here and on their own communications channels.
- nknighthb 13y ago
  I will agree not to blame corporations for the actions of their employees the moment those employees are individually liable for their actions. Corporations always speak with one voice. If that voice says something wrong or inconsistent, it is not the public's problem, it is the corporation's. No excuses. Ever.

alimoeeny 13y ago

I am not their customer, but I admire their acceptance of their mistake, "The second mistake that we made was not notifying our customers that use the API"

ak217 13y ago

The fact that DigitalOcean have no idea how to secure customer block devices makes me wonder what other security issues they are unaware of.

callesgg 13y ago

Either digital ocean actually has real SSd disks for each vm or something is deeply wrong. A new VM should have a new disk file. When I create a vm I create a vmdk file that represents the disk. When I delete the vm I also delete the vmdk file. If I then create a new vm I would use a new vmdk file. Apparently that is not the way digital ocean does things, but how do they do stuff then?

wmf 13y ago
This was covered in the previous thread: https://news.ycombinator.com/item?id=6983270 https://news.ycombinator.com/item?id=6983270 DO may not be using file-based storage because it is slower.