3 ms·

How your data gets compromised in an IaaS cloud: Vendor tells all

26 points by cloudsigma 16y ago

rworthington 16y ago
Do you guys know what the situation is with GoGrid? I've been using them for about 6 months now but I've not been using encryption. Am I exposed to data leakage in the way you outline in your blog post?
- cloud-geek 16y ago
  Any service that reuses the same physical drive space should be vulnerable unless they either store encrypted or do secure deletion. Having used secure deletion before on dedicated hardware it is really hitting the drive performance. That is why I am guessing vendors don't use secure deletion. Using encryption overall looks more efficient overall.
  - cloudsigma 16y ago
    Secure deletion on physical spindles is extremely resource heavy especially if someone is deleting a very large drive. Storing encrypted first does affect performance but it is generally much less and more predictable (doing a big secure delete on a drive inflicts an immediate and unexpected hit on that part of a storage array). We think encryption is also just a lot more robust. If you don't lay down the data in the first place in readable form on the physical drives, its eliminates a lot of data leakage possibilities. Best wishes, Patrick
  - moe 16y ago
    Sorry, but this is really bordering on FUD here. Have you ever looked at a freshly mounted EBS volume on EC2? It shows all zero's for me. And I'm almost sure that was not just a coincidence for the volumes I looked at. Moreover there are more efficient ways than encryption or "full sweep overwrite" to address this at the storage-level.
    - cloudsigma 16y ago
      Our post is regarding IaaS clouds in general and has nothing specifically to do with EC2 or any other particular vendor. It may be the case that EC2 does have secure measures in place; I wonder how many of their customers can articulate them (or customers of other IaaS clouds for that matter)? We are merely raising an important issue. You can look at the many other posts we have on the other various aspects of security in the cloud of which this post forms the latest instalment regarding data storage. "there are more efficient ways than encryption or "full sweep overwrite" to address this at the storage-level." Your suggestion would be? Kind regards, Patrick
      - cloudsigma 16y ago
        Vukk, The issue there is more about tracking. You'd have to track every single block and return zero for any that hadn't been used/altered since drive creation. I'm guessing it would prove pretty costly in terms of latency for drive access after initial drive creation but its a good idea potentially. I'll pass it onto our technical guys as well to ask about its feasibility. Best wishes, Patrick
      - moe 16y ago
        Your suggestion would be? See my reply above about using a loopback-file. It's a one-liner: dd if=/dev/zero of=/vols/customer123/vol234 seek=$SIZE bs=1 count=1 There's surely quite a few other ways to skin this cat (eventually arriving at the multi-tenancy features in proprietary SAN appliances), but this seems the most obvious one.
      - vukk 16y ago
        Since we are not directly accessing the HDD, couldn't the virtualization layer say if(disk block not written before) return 0; and thus the customer wouldn't have to worry? Or is this too inefficient?
patio11 16y ago
This is a big issue in certain verticals. In my early research for AR I looked into the interaction of HIPAA (an American privacy law for medical information) and cloud hosting. My brief educated layperson's conclusion: sensible default settings at your cloud service of choice almist certainly lead you to be OMGWTF noncompliant. I immediately moved medical providers out of scope, because it looked like there were, minimally, several months of engineer time needed to merit a finding of compliance, plus whatever costs/effort it would take to deal with the lawyers.
- cloudsigma 16y ago
  A lot of the compliance issues come from the mixing of infrastructure and the software and networking layers with many IaaS providers. In our cloud only the customer has root access and file system visibility. Essentially the cloud vendor then needs to demonstrate compliance with physical access and data protection/data leakage areas as employees don't have the ability to view data. This isn't a typical situation and for most clouds that are more like IaaS/PaaS hybrids it opens quite a can of worms. As a Swiss based cloud currently we'd be excluded from many US industry sectors which required domestic hosting. This will change shortly (can't say more) and when it does we'll be working to put in place the necessary coverage/compliance certificates to expand into these sectors. Best wishes, Patrick
- samd 16y ago
  My layman's conclusion of HIPAA and cloud hosting was that the language of the law was vague and technologically out of date, and that depending on how you interpreted its requirements you'd be either OK or OMGWTF noncompliant. It's too bad because the medical space is in dire need of innovation and there's a lot of money to be made.
  - cloudsigma 16y ago
    There's definitely huge opportunities in the medical sphere. We do have customers from this sector in our cloud although how they handle client data is always very strictly controlled and I think its fair to say that only a subset of the potential is being realised properly in terms of the broader market. Kind regards, Patrick
trotsky 16y ago
I don't understand why a zero wipe isn't sufficient when provisioning the storage. At least for this purpose it would seem to achieve the same result as encryption with much less complexity and no ongoing overhead. AWS takes a long time to provision new EBS storage, does anyone know what's going on there?
- cloudsigma 16y ago
  Simply overwriting with zeros isn't a 100% fix by any means and actually it does involve writing perhaps 1TB of data in sequence. For any storage array that's a lot of data to write. It doesn't matter if its 0s or something else. Better is to use random data but even so you still have forensic techniques to revert this. You say encryption is complicated but actually as a vendor its implicit in our system. As a customer you just mark the drive upon creation and then its invisible to both you as a customer and the cloud servers that are using the drive. That's really the whole idea, to make security measures that are convenient so people actually use them. Our customers see usually about a 10%-15% performance difference and we've got pretty high storage performance to begin with so it rarely means taking a performance hit compared with other platforms. We also allow multiple drives so users can categorise data by drives for encryption or not. Of course I'm biased regarding performance but the principles stand. The other point of the blog post is to ask; what do other vendors do and do their customers have the ability to find out? Security through obscurity isn't an acceptable approach in the cloud. Everyone needs to be transparent and work to build confidence through solid information and education on how to use the cloud securely and effectively. Kind regards, Patrick
  - moe 16y ago
    Better is to use random data but even so you still have forensic techniques to revert this. How does one revert this without physical access to the drive? We overwrite our EBS volumes with zeros before deleting them and I was under the impression that should protect us against leakage to other customers. Naturally it can't protect us against a malicious amazon employee pulling the drive physically (or taking snapshots without our knowledge), but frankly I don't see how your "vendor encryption" helps with that either. If all I do is set a checkbox to "mark the drive for encryption" then that means a) You have the encryption key, I don't. b) The checkbox could just as well be a placebo and not have any effect. Thus we're back to square 1 and the same old question: "Do I trust you?" No need to take a 10%-15% performance hit for that.
notmyname 16y ago
FWIW, non-block storage services (like Rackspace Cloud Files and S3) should not be vulnerable to these info leaks. I cannot speak to the S3 backend, but this sort of attack would not be possible with Cloud Files. Of course, the use case is a little different when you don't have access to a block-level device.
- cloudsigma 16y ago
  Yes we believe that's the case too however like you say, object orientated storage services are a different use case to block-level storage devices. Customers running their own databases for example can have these data leakage issues if they aren't careful. Best wishes, Patrick

26 points by cloudsigma 16y ago

Do you guys know what the situation is with GoGrid? I've been using them for about 6 months now but I've not been using encryption. Am I exposed to data leakage in the way you outline in your blog post?

cloud-geek 16y ago
Any service that reuses the same physical drive space should be vulnerable unless they either store encrypted or do secure deletion. Having used secure deletion before on dedicated hardware it is really hitting the drive performance. That is why I am guessing vendors don't use secure deletion. Using encryption overall looks more efficient overall.
- cloudsigma 16y ago
  Secure deletion on physical spindles is extremely resource heavy especially if someone is deleting a very large drive. Storing encrypted first does affect performance but it is generally much less and more predictable (doing a big secure delete on a drive inflicts an immediate and unexpected hit on that part of a storage array). We think encryption is also just a lot more robust. If you don't lay down the data in the first place in readable form on the physical drives, its eliminates a lot of data leakage possibilities. Best wishes, Patrick
- moe 16y ago
  Sorry, but this is really bordering on FUD here. Have you ever looked at a freshly mounted EBS volume on EC2? It shows all zero's for me. And I'm almost sure that was not just a coincidence for the volumes I looked at. Moreover there are more efficient ways than encryption or "full sweep overwrite" to address this at the storage-level.
  - cloudsigma 16y ago
    Our post is regarding IaaS clouds in general and has nothing specifically to do with EC2 or any other particular vendor. It may be the case that EC2 does have secure measures in place; I wonder how many of their customers can articulate them (or customers of other IaaS clouds for that matter)? We are merely raising an important issue. You can look at the many other posts we have on the other various aspects of security in the cloud of which this post forms the latest instalment regarding data storage. "there are more efficient ways than encryption or "full sweep overwrite" to address this at the storage-level." Your suggestion would be? Kind regards, Patrick
    - cloudsigma 16y ago
      Vukk, The issue there is more about tracking. You'd have to track every single block and return zero for any that hadn't been used/altered since drive creation. I'm guessing it would prove pretty costly in terms of latency for drive access after initial drive creation but its a good idea potentially. I'll pass it onto our technical guys as well to ask about its feasibility. Best wishes, Patrick
    - moe 16y ago
      Your suggestion would be? See my reply above about using a loopback-file. It's a one-liner: dd if=/dev/zero of=/vols/customer123/vol234 seek=$SIZE bs=1 count=1 There's surely quite a few other ways to skin this cat (eventually arriving at the multi-tenancy features in proprietary SAN appliances), but this seems the most obvious one.
    - vukk 16y ago
      Since we are not directly accessing the HDD, couldn't the virtualization layer say if(disk block not written before) return 0; and thus the customer wouldn't have to worry? Or is this too inefficient?

patio11 16y ago

This is a big issue in certain verticals. In my early research for AR I looked into the interaction of HIPAA (an American privacy law for medical information) and cloud hosting. My brief educated layperson's conclusion: sensible default settings at your cloud service of choice almist certainly lead you to be OMGWTF noncompliant. I immediately moved medical providers out of scope, because it looked like there were, minimally, several months of engineer time needed to merit a finding of compliance, plus whatever costs/effort it would take to deal with the lawyers.

cloudsigma 16y ago
A lot of the compliance issues come from the mixing of infrastructure and the software and networking layers with many IaaS providers. In our cloud only the customer has root access and file system visibility. Essentially the cloud vendor then needs to demonstrate compliance with physical access and data protection/data leakage areas as employees don't have the ability to view data. This isn't a typical situation and for most clouds that are more like IaaS/PaaS hybrids it opens quite a can of worms. As a Swiss based cloud currently we'd be excluded from many US industry sectors which required domestic hosting. This will change shortly (can't say more) and when it does we'll be working to put in place the necessary coverage/compliance certificates to expand into these sectors. Best wishes, Patrick
samd 16y ago
My layman's conclusion of HIPAA and cloud hosting was that the language of the law was vague and technologically out of date, and that depending on how you interpreted its requirements you'd be either OK or OMGWTF noncompliant. It's too bad because the medical space is in dire need of innovation and there's a lot of money to be made.
- cloudsigma 16y ago
  There's definitely huge opportunities in the medical sphere. We do have customers from this sector in our cloud although how they handle client data is always very strictly controlled and I think its fair to say that only a subset of the potential is being realised properly in terms of the broader market. Kind regards, Patrick

trotsky 16y ago

I don't understand why a zero wipe isn't sufficient when provisioning the storage. At least for this purpose it would seem to achieve the same result as encryption with much less complexity and no ongoing overhead. AWS takes a long time to provision new EBS storage, does anyone know what's going on there?

cloudsigma 16y ago
Simply overwriting with zeros isn't a 100% fix by any means and actually it does involve writing perhaps 1TB of data in sequence. For any storage array that's a lot of data to write. It doesn't matter if its 0s or something else. Better is to use random data but even so you still have forensic techniques to revert this. You say encryption is complicated but actually as a vendor its implicit in our system. As a customer you just mark the drive upon creation and then its invisible to both you as a customer and the cloud servers that are using the drive. That's really the whole idea, to make security measures that are convenient so people actually use them. Our customers see usually about a 10%-15% performance difference and we've got pretty high storage performance to begin with so it rarely means taking a performance hit compared with other platforms. We also allow multiple drives so users can categorise data by drives for encryption or not. Of course I'm biased regarding performance but the principles stand. The other point of the blog post is to ask; what do other vendors do and do their customers have the ability to find out? Security through obscurity isn't an acceptable approach in the cloud. Everyone needs to be transparent and work to build confidence through solid information and education on how to use the cloud securely and effectively. Kind regards, Patrick
- moe 16y ago
  Better is to use random data but even so you still have forensic techniques to revert this. How does one revert this without physical access to the drive? We overwrite our EBS volumes with zeros before deleting them and I was under the impression that should protect us against leakage to other customers. Naturally it can't protect us against a malicious amazon employee pulling the drive physically (or taking snapshots without our knowledge), but frankly I don't see how your "vendor encryption" helps with that either. If all I do is set a checkbox to "mark the drive for encryption" then that means a) You have the encryption key, I don't. b) The checkbox could just as well be a placebo and not have any effect. Thus we're back to square 1 and the same old question: "Do I trust you?" No need to take a 10%-15% performance hit for that.

notmyname 16y ago

FWIW, non-block storage services (like Rackspace Cloud Files and S3) should not be vulnerable to these info leaks. I cannot speak to the S3 backend, but this sort of attack would not be possible with Cloud Files. Of course, the use case is a little different when you don't have access to a block-level device.

cloudsigma 16y ago
Yes we believe that's the case too however like you say, object orientated storage services are a different use case to block-level storage devices. Customers running their own databases for example can have these data leakage issues if they aren't careful. Best wishes, Patrick