4 ms·

AWS S3 beginning to apply 2 security best practices all new buckets by default

54 points by nixcraft 4y ago

hdjjhhvvhga 4y ago
What is funny is that although they have been phasing out S3 ACLs for years, they are still using it for their own products. For example, Control Tower uses S3 ACLs to secure access to S3 buckets with logs.
based2 4y ago
competition on SAML authn: * https://learn.microsoft.com/en-us/azure/active-directory/manage-apps/howto-enforce-signed-saml-authentication https://learn.microsoft.com/en-us/azure/active-directory/man... * https://www.alibabacloud.com/help/en/resource-access-management/latest/saml-response-for-role-based-sso https://www.alibabacloud.com/help/en/resource-access-managem...
nutbear 4y ago
I wrote a blog detailing what this change means on S3 ACLs and Block Public Access on by default: https://www.cloudquery.io/blog/finding-enabled-s3-acls-and-disabled-s3-block-public-access https://www.cloudquery.io/blog/finding-enabled-s3-acls-and-d...
drewda 4y ago
Given how many major data breaches have been the result of unintentional public access to orgaanizations' data on S3, I almost think Amazon should remove all public access to buckets and objects from the entire S3 product. Instead make all access to S3 be through credentialed access or signed URLs. If users need to expose an entire bucket to the public Internet, make them go to the effort to put a service in front of the bucket. Yes, this would be a huge change. But playing with the default values for S3 seems like too little, too late.
- geoduck14 4y ago
  Please don't do this. I've used S3 to host static web pages and adding an additional service would add complexity and cost. Also, I'll point out that even a couple of years ago, AWS made it rather uncomfortable for me to expose pages to the public. IIRC, each time I changed the html, I needed to also remind AWS that I wanted the file public.
  - x3n0ph3n3 4y ago
    Just put CloudFront in front of it. Then you get to use your own domain.
    - master_crab 4y ago
      You dont need Cloudfront to use your own domain. You can do it with buckets alone too, but they have to follow a clear naming policy.
      - x3n0ph3n3 4y ago
        You do if you want HTTPS... > Amazon S3 does not support HTTPS access to the website. If you want to use HTTPS, you can use Amazon CloudFront to serve a static website hosted on Amazon S3.
    - laserlight 4y ago
      Using your own domain doesn't require CloudFront [0]. [0] https://docs.aws.amazon.com/AmazonS3/latest/userguide/website-hosting-custom-domain-walkthrough.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/websit...
      - x3n0ph3n3 4y ago
        It does if you want HTTPS. > Amazon S3 does not support HTTPS access to the website. If you want to use HTTPS, you can use Amazon CloudFront to serve a static website hosted on Amazon S3.
- mjr00 4y ago
  AWS already makes it a massive headache with all sorts of warning flags if you try to make an object public, and this change will make it even more obvious. S3 is great for hosting static websites for pennies a month and zero work required to configure nginx etc. There's a huge use case for allowing public access to data in S3 buckets.
- master_crab 4y ago
  The initial default behavior for S3 was public access. Now requiring already created buckets to have default private-read will probably - almost literally and with no hint of sarcasm - break the internet. New buckets, sure whatever, but dont touch the old stuff.
  - based2 4y ago
    irc -> mastodon pop -> pop tls ldap -> ldaps ftp -> scp http -> https (ssl -> tls)
- mvanbaak 4y ago
  Please no. Having S3 (with or without cloud front) being able to be your 'serverless' webserver is one of the very strong points of the service. Before putting important info on the machines of a cloud provider, hire people that know how the service works. Yes, this costs money, but the 'solution' you are providing in your comment as well, and can be equally open and leaking. Specially if the current team can not secure a bucket, they will not be able to secure a proxy in front of it.
  - jeffalyanak 4y ago
    You could still do that with Cloudfront. It's simple to configure but makes it a deliberate action and not something that could be done accidentally.
    - mvanbaak 4y ago
      Have you seen what you have to do to make a bucket public nowadays? It's not as if you do it by accident (there's a couple of warnings etc) Yes, buckets created with the API dont have public access blocked, so if someone that does not fully grasp cloud security is given access to create buckets ... Luckily this is now fixed with the latest round of changes done by AWS.
    - simplotek 4y ago
      > You could still do that with Cloudfront. You can also expect users to audit what they do with S3, which is something they should already be doing, and that does not force anyone to suddenly refactor their whole deployment workflow. It makes absolutely no sense at all to argue that public access to S3 should be shut off and those who use it should start paying for CloudFront just because you feel that its too hard to check if your bucket is set to grant public access. Also, there's already AWS Config for those who feel they need guardrails to enforce a specific configuration. AWS Config even let's you put together a Lambda to switch off public access to a bucket if some sloppy fingers indavertentlh set it on. https://aws.amazon.com/config/ https://aws.amazon.com/config/ It boggles the mind how anyone could suggest with a straight face that your personal usecases should be automated away even though it screws over everyone else using it.
- simplotek 4y ago
  > If users need to expose an entire bucket to the public Internet, make them go to the effort to put a service in front of the bucket. This would defeat the purpose of using S3 to serve static content, which is one of the main usecases for S3.
- vlovich123 4y ago
  Have you looked at R2? You don’t need to put a full service in front. You can set up any custom domain through Cloudflare you want in front of it and then manage access policies through the zone and Access. It would be nice for the default to be when you make it public, all objects are still inaccessible until you set up explicit policies (ie allow all access or just to specific objects). That may be too onerous though as the most common use case appears to be making entire buckets public. You want it easy to follow good security policies without making it a hoop jumping exercise. It’s a difficult problem (not sure why it took them so long to make this particular change though). You can of course make things public via a secret managed r2.dev URL (it’s a new UUID every time you make a public bucket) for testing and comparing against access via your zone if debugging. But we discourage it slightly in the first place (if I recall correctly it’s a more hidden/demphasized option in the UI flow for setting up a public access) as it’s really only intended for testing as it’s a managed service and we may make functional changes to it’s behavior at any time. I’m not trying to crap on S3 or anything. They have a much older codebase and larger number of customers to deal with. I’m just highlighting you can recognize that public buckets are an extremely common use case and it’s possible to do better I think without adding a lot of complexity. Disclaimer: worked on R2
- Waterluvian 4y ago
  S3 seems to be one of the most entry level AWS products. I see non-engineers using it all the time. I think this screams, “these people need to be saved from themselves!” But it also suggests why S3 retains so many features that are more “properly” done through a suite of other cloud products.
- Turing_Machine 4y ago
  Not needing to run a separate service is pretty much the entire reason I use S3. I've got some low-traffic web sites on there that "just work". I don't have to screw around with keeping Apache or Express or some other web server up to date or otherwise managing them, nor do I have to worry about the sites either a) falling over from an unusual traffic spike or b) massively overprovisioning them to prevent them from falling over from an unusual traffic spike.
rcme 4y ago
What is the alternative to ACLs? Or is reading from users / roles in the same project supported by default, provided the user / role has the required permissions?
- x3n0ph3n3 4y ago
  Policies are the alternative to ACLs. You craft your IAM user/role policies or your bucket's resource policy to allow and deny the appropriate access. ACLs are a totally separate subsystem of S3 that should be been deprecated long ago.
leetrout 4y ago
They also default to encrypting objects now. https://aws.amazon.com/blogs/aws/amazon-s3-encrypts-new-objects-by-default/ https://aws.amazon.com/blogs/aws/amazon-s3-encrypts-new-obje...
- c7DJTLrn 4y ago
  I'm a Platform Engineer but I've never actually understood (or really looked into) what they mean by encryption in this context. I checked that link and they call it "server-side encryption" -- do they mean encryption at rest? If so, it seems terrible that they would allow data to live unencrypted on their non-volatile storage, why do they even give the customer that option? Other than for very specific kinds of customers that want to supply their own encryption keys, why should it even be their concern? The whole point of cloud is to take the hardware away. Encryption is cheap, just do it.
  - leetrout 4y ago
    Yes it is just encryption at rest and their API access mechanisms automatically apply the encrypt or decrypt action on your behalf. If your access is leaky encryption doesnt buy you anything. I take special offense to teams who store Terraform state in S3 and claim "it's encrypted" when it is so easy for other users in the same AWS account can easily access the buckets contents. GCS is slightly more secure but you really need client side encryption to be safest.
    - kclay 4y ago
      May I ask what is the proper way to store Terraform state. We are currently testing out Terraform at my job and it just uses a s3 bucket with exception turned on. Thanks
      - leetrout 4y ago
        Oh we do, too. My beef with it is how easy it is for a user on the account to go and read your state due to a lax IAM or bucket policy. My advice: check and make sure your bucket policy you use for the state has an explicit deny (resource *, principal *) and then you explicitly allow only the user / role that requires access to the TF state. Things to watch out for are providers that store sensitive info in your state. For example, if you use Vault and you read a secret out of Vault with Terraform then the secret will be saved in your Terraform state which, painting with broad strokes, largely invalidates the purpose of Vault. Lots of providers do this, some are getting better about not requiring sensitive info to be saved in the state or included in the config.
ed25519FUUU 4y ago
Thankfully I’ve never been charged with keeping any serious PII in an S3 bucket, because the permissions have always worried me, and I’d probably be considered an expert with IAM policies. Thankfully with S3 it’s getting easier and easier to do the right thing. I’m glad for the S3 team moving in this direction.

54 points by nixcraft 4y ago

What is funny is that although they have been phasing out S3 ACLs for years, they are still using it for their own products. For example, Control Tower uses S3 ACLs to secure access to S3 buckets with logs.

based2 4y ago

competition on SAML authn: * https://learn.microsoft.com/en-us/azure/active-directory/manage-apps/howto-enforce-signed-saml-authentication https://learn.microsoft.com/en-us/azure/active-directory/man... * https://www.alibabacloud.com/help/en/resource-access-management/latest/saml-response-for-role-based-sso https://www.alibabacloud.com/help/en/resource-access-managem...

nutbear 4y ago

I wrote a blog detailing what this change means on S3 ACLs and Block Public Access on by default: https://www.cloudquery.io/blog/finding-enabled-s3-acls-and-disabled-s3-block-public-access https://www.cloudquery.io/blog/finding-enabled-s3-acls-and-d...

drewda 4y ago

Given how many major data breaches have been the result of unintentional public access to orgaanizations' data on S3, I almost think Amazon should remove all public access to buckets and objects from the entire S3 product. Instead make all access to S3 be through credentialed access or signed URLs. If users need to expose an entire bucket to the public Internet, make them go to the effort to put a service in front of the bucket. Yes, this would be a huge change. But playing with the default values for S3 seems like too little, too late.

geoduck14 4y ago
Please don't do this. I've used S3 to host static web pages and adding an additional service would add complexity and cost. Also, I'll point out that even a couple of years ago, AWS made it rather uncomfortable for me to expose pages to the public. IIRC, each time I changed the html, I needed to also remind AWS that I wanted the file public.
- x3n0ph3n3 4y ago
  Just put CloudFront in front of it. Then you get to use your own domain.
  - master_crab 4y ago
    You dont need Cloudfront to use your own domain. You can do it with buckets alone too, but they have to follow a clear naming policy.
    - x3n0ph3n3 4y ago
      You do if you want HTTPS... > Amazon S3 does not support HTTPS access to the website. If you want to use HTTPS, you can use Amazon CloudFront to serve a static website hosted on Amazon S3.
  - laserlight 4y ago
    Using your own domain doesn't require CloudFront [0]. [0] https://docs.aws.amazon.com/AmazonS3/latest/userguide/website-hosting-custom-domain-walkthrough.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/websit...
    - x3n0ph3n3 4y ago
      It does if you want HTTPS. > Amazon S3 does not support HTTPS access to the website. If you want to use HTTPS, you can use Amazon CloudFront to serve a static website hosted on Amazon S3.
mjr00 4y ago
AWS already makes it a massive headache with all sorts of warning flags if you try to make an object public, and this change will make it even more obvious. S3 is great for hosting static websites for pennies a month and zero work required to configure nginx etc. There's a huge use case for allowing public access to data in S3 buckets.
master_crab 4y ago
The initial default behavior for S3 was public access. Now requiring already created buckets to have default private-read will probably - almost literally and with no hint of sarcasm - break the internet. New buckets, sure whatever, but dont touch the old stuff.
- based2 4y ago
  irc -> mastodon pop -> pop tls ldap -> ldaps ftp -> scp http -> https (ssl -> tls)
mvanbaak 4y ago
Please no. Having S3 (with or without cloud front) being able to be your 'serverless' webserver is one of the very strong points of the service. Before putting important info on the machines of a cloud provider, hire people that know how the service works. Yes, this costs money, but the 'solution' you are providing in your comment as well, and can be equally open and leaking. Specially if the current team can not secure a bucket, they will not be able to secure a proxy in front of it.
- jeffalyanak 4y ago
  You could still do that with Cloudfront. It's simple to configure but makes it a deliberate action and not something that could be done accidentally.
  - mvanbaak 4y ago
    Have you seen what you have to do to make a bucket public nowadays? It's not as if you do it by accident (there's a couple of warnings etc) Yes, buckets created with the API dont have public access blocked, so if someone that does not fully grasp cloud security is given access to create buckets ... Luckily this is now fixed with the latest round of changes done by AWS.
  - simplotek 4y ago
    > You could still do that with Cloudfront. You can also expect users to audit what they do with S3, which is something they should already be doing, and that does not force anyone to suddenly refactor their whole deployment workflow. It makes absolutely no sense at all to argue that public access to S3 should be shut off and those who use it should start paying for CloudFront just because you feel that its too hard to check if your bucket is set to grant public access. Also, there's already AWS Config for those who feel they need guardrails to enforce a specific configuration. AWS Config even let's you put together a Lambda to switch off public access to a bucket if some sloppy fingers indavertentlh set it on. https://aws.amazon.com/config/ https://aws.amazon.com/config/ It boggles the mind how anyone could suggest with a straight face that your personal usecases should be automated away even though it screws over everyone else using it.
simplotek 4y ago
> If users need to expose an entire bucket to the public Internet, make them go to the effort to put a service in front of the bucket. This would defeat the purpose of using S3 to serve static content, which is one of the main usecases for S3.
vlovich123 4y ago
Have you looked at R2? You don’t need to put a full service in front. You can set up any custom domain through Cloudflare you want in front of it and then manage access policies through the zone and Access. It would be nice for the default to be when you make it public, all objects are still inaccessible until you set up explicit policies (ie allow all access or just to specific objects). That may be too onerous though as the most common use case appears to be making entire buckets public. You want it easy to follow good security policies without making it a hoop jumping exercise. It’s a difficult problem (not sure why it took them so long to make this particular change though). You can of course make things public via a secret managed r2.dev URL (it’s a new UUID every time you make a public bucket) for testing and comparing against access via your zone if debugging. But we discourage it slightly in the first place (if I recall correctly it’s a more hidden/demphasized option in the UI flow for setting up a public access) as it’s really only intended for testing as it’s a managed service and we may make functional changes to it’s behavior at any time. I’m not trying to crap on S3 or anything. They have a much older codebase and larger number of customers to deal with. I’m just highlighting you can recognize that public buckets are an extremely common use case and it’s possible to do better I think without adding a lot of complexity. Disclaimer: worked on R2
Waterluvian 4y ago
S3 seems to be one of the most entry level AWS products. I see non-engineers using it all the time. I think this screams, “these people need to be saved from themselves!” But it also suggests why S3 retains so many features that are more “properly” done through a suite of other cloud products.
Turing_Machine 4y ago
Not needing to run a separate service is pretty much the entire reason I use S3. I've got some low-traffic web sites on there that "just work". I don't have to screw around with keeping Apache or Express or some other web server up to date or otherwise managing them, nor do I have to worry about the sites either a) falling over from an unusual traffic spike or b) massively overprovisioning them to prevent them from falling over from an unusual traffic spike.

rcme 4y ago

What is the alternative to ACLs? Or is reading from users / roles in the same project supported by default, provided the user / role has the required permissions?

x3n0ph3n3 4y ago
Policies are the alternative to ACLs. You craft your IAM user/role policies or your bucket's resource policy to allow and deny the appropriate access. ACLs are a totally separate subsystem of S3 that should be been deprecated long ago.

leetrout 4y ago

They also default to encrypting objects now. https://aws.amazon.com/blogs/aws/amazon-s3-encrypts-new-objects-by-default/ https://aws.amazon.com/blogs/aws/amazon-s3-encrypts-new-obje...

c7DJTLrn 4y ago
I'm a Platform Engineer but I've never actually understood (or really looked into) what they mean by encryption in this context. I checked that link and they call it "server-side encryption" -- do they mean encryption at rest? If so, it seems terrible that they would allow data to live unencrypted on their non-volatile storage, why do they even give the customer that option? Other than for very specific kinds of customers that want to supply their own encryption keys, why should it even be their concern? The whole point of cloud is to take the hardware away. Encryption is cheap, just do it.
- leetrout 4y ago
  Yes it is just encryption at rest and their API access mechanisms automatically apply the encrypt or decrypt action on your behalf. If your access is leaky encryption doesnt buy you anything. I take special offense to teams who store Terraform state in S3 and claim "it's encrypted" when it is so easy for other users in the same AWS account can easily access the buckets contents. GCS is slightly more secure but you really need client side encryption to be safest.
  - kclay 4y ago
    May I ask what is the proper way to store Terraform state. We are currently testing out Terraform at my job and it just uses a s3 bucket with exception turned on. Thanks
    - leetrout 4y ago
      Oh we do, too. My beef with it is how easy it is for a user on the account to go and read your state due to a lax IAM or bucket policy. My advice: check and make sure your bucket policy you use for the state has an explicit deny (resource *, principal *) and then you explicitly allow only the user / role that requires access to the TF state. Things to watch out for are providers that store sensitive info in your state. For example, if you use Vault and you read a secret out of Vault with Terraform then the secret will be saved in your Terraform state which, painting with broad strokes, largely invalidates the purpose of Vault. Lots of providers do this, some are getting better about not requiring sensitive info to be saved in the state or included in the config.

ed25519FUUU 4y ago

Thankfully I’ve never been charged with keeping any serious PII in an S3 bucket, because the permissions have always worried me, and I’d probably be considered an expert with IAM policies. Thankfully with S3 it’s getting easier and easier to do the right thing. I’m glad for the S3 team moving in this direction.