7 ms·

Amazon S3 now automatically encrypts all new objects

73 points by redbell 4y ago

scrollaway 4y ago
That’s awesome. Zero cost and performance drop, too. S3 is frankly amazing technology; probably the most robust and well rounded piece of cloud tech that exists.
deleted 4y ago
[deleted]
javaunsafe2019 4y ago
I really would love to know how s3 is implemented. Is it maybe just using a backed up file system and some api?
- andyp-kw 4y ago
  I read that DynamoDB and S3 share many similarities in terms of architecture.
- simonw 4y ago
  There's some good discussion of S3 internals in this thread from 2021: https://news.ycombinator.com/item?id=26466338 https://news.ycombinator.com/item?id=26466338
  - javaunsafe2019 4y ago
    Thank you
- deleted 4y ago
  [deleted]
simonw 4y ago
I'm so impressed at how S3 can roll out changes like this on a regularly frequent basis that impact an unimaginably large number of servers and stored data. This is an incredible piece of engineering. Almost as impressive as when they made S3 read-after-write consistent a couple of years ago: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/ https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...
- daenney 4y ago
  The only thing impressive about this is how long it’s taken them to make this the default. This has been available since 2011 and other cloud providers have had encryption at rest by default for years. The read-after-write consistency was definitely an impressive change. This however, not so much.
  - Genbox 4y ago
    Encryption enabled by default at different S3 compatible providers: - [x] Microsoft Azure (https://learn.microsoft.com/en-us/azure/storage/common/storage-service-encryption https://learn.microsoft.com/en-us/azure/storage/common/stora...) - [x] Google Cloud Storage (https://cloud.google.com/storage/docs/encryption https://cloud.google.com/storage/docs/encryption) - [x] Backblaze B2 (https://help.backblaze.com/hc/en-us/articles/217973398-Encrypting-Files-in-B2 https://help.backblaze.com/hc/en-us/articles/217973398-Encry...) - [x] Wasabi (https://wasabi-support.zendesk.com/hc/en-us/articles/115001693992-How-secure-is-my-data- https://wasabi-support.zendesk.com/hc/en-us/articles/1150016...) - [ ] DigitalOcean (https://www.digitalocean.com/community/tutorials/how-to-create-an-encrypted-file-system-on-a-digitalocean-block-storage-volume https://www.digitalocean.com/community/tutorials/how-to-crea...) - [ ] DreamHost (https://help.dreamhost.com/hc/en-us/articles/360001089163-Encrypting-DreamObjects-data https://help.dreamhost.com/hc/en-us/articles/360001089163-En...) - [ ] Linode (https://www.linode.com/docs/guides/server-side-encryption/ https://www.linode.com/docs/guides/server-side-encryption/)
regecks 4y ago
Is Amazon-managed encryption (SSE-S3) mostly a checklist/regulatory thing? Like, is it mainly protecting against somebody walking out of a DC with a storage device?
- hbogert 4y ago
  Yeah this question. I'm not well versed in Amazon lingo to understand the article
- tgv 4y ago
  Or some other kind of hack/data theft. That's what it looks like to me. I don't see it as just a checklist thing: if Amazon ever gets hacked, you wouldn't want your data compromised as a result. Of course, if they manage to steal the keys too, it would still be.
- ororroro 4y ago
  Yes. This is really a non-announcement. S3 has had configurable encryption since at least 2011. And Google cloud storage has had default encryption at rest since 2013. For a decent explanation of encryption at rest check out https://cloud.google.com/docs/security/encryption/default-encryption https://cloud.google.com/docs/security/encryption/default-en...
- dividuum 4y ago
  I would have assumed that's always the case for all storage devices? Or am I naive? I would expected any cloud provider to do that, even if it's just to make disposing old drives easier.
awoimbee 4y ago
What they won't tell you is that this will use the default KMS key, which cannot be shared across accounts. So your S3 bucket objects cannot be shared across AWS accounts by default, unless you setup a custom KMS key and "copy" everything in place (a standard AWS organisation is made of multiple accounts).
- paranoidrobot 4y ago
  The first line of the article reads otherwise, that it's using the AES256 option, which does work cross-account without KMS permissions. > Amazon S3 now automatically applies S3 managed server-side encryption (SSE-S3) as a base level of encryption to all new objects added to S3
- orf 4y ago
  That's not correct, this isn't defaulting to SSE-KMS. and it's trivial to share KMS encrypted objects cross-account.
kedihacker 4y ago
Did files hit the disk unencrypted before this or there were full disk encryption?
onphonenow 4y ago
Does anyone have rough numbers on s3 scale? Most storage classes have redundant copies as well so 3x things maybe from that. YouTube / S3 - a few services seem like they must have crazy storage scale
- majewsky 4y ago
  If you're building one of these object storages for yourself (say, using Ceph or OpenStack Swift), a plausible configuration is 12 drives with 8 TB each, or 96 TB per node [1]. Let's round that to 100 TB for simplicity. You obviously want to spread your replicas across multiple datacenters, so let's just say you'll put 10 of these servers (i.e. around half a rack full) each into three different datacenters. That's a very modest deployment in terms of floor space, and yet already has a capacity in the low petabyte range (3 PB raw capacity, 1 PB net capacity). On the AWS scale, object storage servers will probably be filling entire datacenters (or at least an appreciable portion of each datacenter), so it's very easy to see them being well into the exabyte (= million terabyte) scale, though probably not zettabyte (= billion terabyte) yet. [1] Source: I help maintain an object storage for a private cloud. Disclaimer: These are not the exact numbers for our nodes, but in the same ballpark.
  - QuinnyPig 4y ago
    Erasure coding drops the drive requirements massively; they talked about it at reinvent 2021.
- oxymoron 4y ago
  There is a talk by the S3 VP on Youtube which mentions some rough numbers, I think it’s from re:invent 2019. Also, they mention 100 trillion objects here https://aws.amazon.com/blogs/aws/amazon-s3s-15th-birthday-it-is-still-day-1-after-5475-days-100-trillion-objects/ https://aws.amazon.com/blogs/aws/amazon-s3s-15th-birthday-it...
funstuff007 4y ago
What does encryption at rest protect against with a Cloud provider? I assume the read credentials must ipso facto decrypt the object. So encryption at rest protects you against an inside job at AWS (smash and grab drives) or government confiscation. Am I thinking about this correctly?
- QuadrupleA 4y ago
  Yeah - physical drive theft I think. I assume that's well locked down (and a crap shoot whose data someone'd get if they stole some drives) but nice to be protected I guess.
  - voakbasda 4y ago
    I am thinking that it lowers their cost for recycling drives. If everything is encrypted at rest, you don’t need to go to crazy lengths to wipe the drives.
    - majewsky 4y ago
      This is it. I help maintain the object storage for a private cloud. Because we have LUKS, preparing servers for decommissioning is done in a matter of minutes (once the servers have been removed from the cluster configuration, of course). We only have to stop all server processes, unmount the disks, overwrite the LUKS headers with /dev/random. Then server management kicks the nodes out of the Kubernetes cluster, then data center management takes care of the hardware.
  - oxymoron 4y ago
    When I started working for AWS as an SDE, I was hoping it’d be possible to visit a datacenter. I was surprised to find out that I’d require L11 (!) approval to so so! The only L11 in my reporting chain is Adam Selipsky. I’m told the AWS data centers has red zones, which no harddrive can be taken out of, without being mechanically and violently destroyed first.
- betaby 4y ago
  How it's protects from government confiscation if key is also stored at AWS (most likely on a different server)? Please explain.
mdaniel 4y ago
Some light previous discussion, although a lot of it was because that specific article was hot garbage: https://news.ycombinator.com/item?id=34276986 https://news.ycombinator.com/item?id=34276986
betaby 4y ago
I read in the past, yet it's still unclear to me. It is a symmetric encryption, right? And as I understand keys are stored at AWS as well. Am I missing something?
lambo4bkfast 4y ago
Any insight into how this is done without impacting performance for reads as they now need to decrypt the data?
- twoodfin 4y ago
  I’d guess that the top tier of S3 physical storage is all SSD, and hardware-driven encryption on such devices is virtually free, as well as desirable for lifecycle reasons. To allow this to double as ”user level” encryption you need to coordinate & manage the keys used vs. just picking something random when the drive is formatted. This is how Apple’s and others’ full-disk encryption has worked for years.

73 points by redbell 4y ago

That’s awesome. Zero cost and performance drop, too. S3 is frankly amazing technology; probably the most robust and well rounded piece of cloud tech that exists.

deleted 4y ago

[deleted]

javaunsafe2019 4y ago

I really would love to know how s3 is implemented. Is it maybe just using a backed up file system and some api?

andyp-kw 4y ago
I read that DynamoDB and S3 share many similarities in terms of architecture.
simonw 4y ago
There's some good discussion of S3 internals in this thread from 2021: https://news.ycombinator.com/item?id=26466338 https://news.ycombinator.com/item?id=26466338
- javaunsafe2019 4y ago
  Thank you
deleted 4y ago
[deleted]

simonw 4y ago

I'm so impressed at how S3 can roll out changes like this on a regularly frequent basis that impact an unimaginably large number of servers and stored data. This is an incredible piece of engineering. Almost as impressive as when they made S3 read-after-write consistent a couple of years ago: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/ https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...

daenney 4y ago
The only thing impressive about this is how long it’s taken them to make this the default. This has been available since 2011 and other cloud providers have had encryption at rest by default for years. The read-after-write consistency was definitely an impressive change. This however, not so much.
- Genbox 4y ago
  Encryption enabled by default at different S3 compatible providers: - [x] Microsoft Azure (https://learn.microsoft.com/en-us/azure/storage/common/storage-service-encryption https://learn.microsoft.com/en-us/azure/storage/common/stora...) - [x] Google Cloud Storage (https://cloud.google.com/storage/docs/encryption https://cloud.google.com/storage/docs/encryption) - [x] Backblaze B2 (https://help.backblaze.com/hc/en-us/articles/217973398-Encrypting-Files-in-B2 https://help.backblaze.com/hc/en-us/articles/217973398-Encry...) - [x] Wasabi (https://wasabi-support.zendesk.com/hc/en-us/articles/115001693992-How-secure-is-my-data- https://wasabi-support.zendesk.com/hc/en-us/articles/1150016...) - [ ] DigitalOcean (https://www.digitalocean.com/community/tutorials/how-to-create-an-encrypted-file-system-on-a-digitalocean-block-storage-volume https://www.digitalocean.com/community/tutorials/how-to-crea...) - [ ] DreamHost (https://help.dreamhost.com/hc/en-us/articles/360001089163-Encrypting-DreamObjects-data https://help.dreamhost.com/hc/en-us/articles/360001089163-En...) - [ ] Linode (https://www.linode.com/docs/guides/server-side-encryption/ https://www.linode.com/docs/guides/server-side-encryption/)

regecks 4y ago

Is Amazon-managed encryption (SSE-S3) mostly a checklist/regulatory thing? Like, is it mainly protecting against somebody walking out of a DC with a storage device?

hbogert 4y ago
Yeah this question. I'm not well versed in Amazon lingo to understand the article
tgv 4y ago
Or some other kind of hack/data theft. That's what it looks like to me. I don't see it as just a checklist thing: if Amazon ever gets hacked, you wouldn't want your data compromised as a result. Of course, if they manage to steal the keys too, it would still be.
ororroro 4y ago
Yes. This is really a non-announcement. S3 has had configurable encryption since at least 2011. And Google cloud storage has had default encryption at rest since 2013. For a decent explanation of encryption at rest check out https://cloud.google.com/docs/security/encryption/default-encryption https://cloud.google.com/docs/security/encryption/default-en...
dividuum 4y ago
I would have assumed that's always the case for all storage devices? Or am I naive? I would expected any cloud provider to do that, even if it's just to make disposing old drives easier.

awoimbee 4y ago

What they won't tell you is that this will use the default KMS key, which cannot be shared across accounts. So your S3 bucket objects cannot be shared across AWS accounts by default, unless you setup a custom KMS key and "copy" everything in place (a standard AWS organisation is made of multiple accounts).

paranoidrobot 4y ago
The first line of the article reads otherwise, that it's using the AES256 option, which does work cross-account without KMS permissions. > Amazon S3 now automatically applies S3 managed server-side encryption (SSE-S3) as a base level of encryption to all new objects added to S3
orf 4y ago
That's not correct, this isn't defaulting to SSE-KMS. and it's trivial to share KMS encrypted objects cross-account.

kedihacker 4y ago

Did files hit the disk unencrypted before this or there were full disk encryption?

onphonenow 4y ago

Does anyone have rough numbers on s3 scale? Most storage classes have redundant copies as well so 3x things maybe from that. YouTube / S3 - a few services seem like they must have crazy storage scale

majewsky 4y ago
If you're building one of these object storages for yourself (say, using Ceph or OpenStack Swift), a plausible configuration is 12 drives with 8 TB each, or 96 TB per node [1]. Let's round that to 100 TB for simplicity. You obviously want to spread your replicas across multiple datacenters, so let's just say you'll put 10 of these servers (i.e. around half a rack full) each into three different datacenters. That's a very modest deployment in terms of floor space, and yet already has a capacity in the low petabyte range (3 PB raw capacity, 1 PB net capacity). On the AWS scale, object storage servers will probably be filling entire datacenters (or at least an appreciable portion of each datacenter), so it's very easy to see them being well into the exabyte (= million terabyte) scale, though probably not zettabyte (= billion terabyte) yet. [1] Source: I help maintain an object storage for a private cloud. Disclaimer: These are not the exact numbers for our nodes, but in the same ballpark.
- QuinnyPig 4y ago
  Erasure coding drops the drive requirements massively; they talked about it at reinvent 2021.
oxymoron 4y ago
There is a talk by the S3 VP on Youtube which mentions some rough numbers, I think it’s from re:invent 2019. Also, they mention 100 trillion objects here https://aws.amazon.com/blogs/aws/amazon-s3s-15th-birthday-it-is-still-day-1-after-5475-days-100-trillion-objects/ https://aws.amazon.com/blogs/aws/amazon-s3s-15th-birthday-it...

funstuff007 4y ago

What does encryption at rest protect against with a Cloud provider? I assume the read credentials must ipso facto decrypt the object. So encryption at rest protects you against an inside job at AWS (smash and grab drives) or government confiscation. Am I thinking about this correctly?

QuadrupleA 4y ago
Yeah - physical drive theft I think. I assume that's well locked down (and a crap shoot whose data someone'd get if they stole some drives) but nice to be protected I guess.
- voakbasda 4y ago
  I am thinking that it lowers their cost for recycling drives. If everything is encrypted at rest, you don’t need to go to crazy lengths to wipe the drives.
  - majewsky 4y ago
    This is it. I help maintain the object storage for a private cloud. Because we have LUKS, preparing servers for decommissioning is done in a matter of minutes (once the servers have been removed from the cluster configuration, of course). We only have to stop all server processes, unmount the disks, overwrite the LUKS headers with /dev/random. Then server management kicks the nodes out of the Kubernetes cluster, then data center management takes care of the hardware.
- oxymoron 4y ago
  When I started working for AWS as an SDE, I was hoping it’d be possible to visit a datacenter. I was surprised to find out that I’d require L11 (!) approval to so so! The only L11 in my reporting chain is Adam Selipsky. I’m told the AWS data centers has red zones, which no harddrive can be taken out of, without being mechanically and violently destroyed first.
betaby 4y ago
How it's protects from government confiscation if key is also stored at AWS (most likely on a different server)? Please explain.

mdaniel 4y ago

Some light previous discussion, although a lot of it was because that specific article was hot garbage: https://news.ycombinator.com/item?id=34276986 https://news.ycombinator.com/item?id=34276986

betaby 4y ago

I read in the past, yet it's still unclear to me. It is a symmetric encryption, right? And as I understand keys are stored at AWS as well. Am I missing something?

lambo4bkfast 4y ago

Any insight into how this is done without impacting performance for reads as they now need to decrypt the data?

twoodfin 4y ago
I’d guess that the top tier of S3 physical storage is all SSD, and hardware-driven encryption on such devices is virtually free, as well as desirable for lifecycle reasons. To allow this to double as ”user level” encryption you need to coordinate & manage the keys used vs. just picking something random when the drive is formatted. This is how Apple’s and others’ full-disk encryption has worked for years.