18 ms·

AWS Support able to access any S3 object due to permission change

450 points by zdw 5y ago

rronalddas 5y ago
KMS Encrypted objects shouldn't be affected though
- cateof 5y ago
  Objects encrypted with S3-managed encryption keys (SSE-S3) are affected, as these keys are set up with a non-configurable resource policy granting the S3 service decryption permissions.
- hericium 5y ago
  Aren't KMS keys created by Amazon?
  - RKearney 5y ago
    Yes but this role did not add the necessary privileges for it to use customer KMS keys. You can’t get an S3 object that’s encrypted with a KMS key if you don’t also have permission to decrypt with that key. Of course Amazon could just give themselves access to decrypt with your KMS keys too, but that didn’t happen here.
  - raffraffraff 5y ago
    There are three types of S3 server side encryption: - SSE-KMS - SSE-S3 - SSE-C Without having an AWS support person test each type and report back, one must assume that the only bulletproof s3 encryption methods are client-side (where you handle encryption and decryption yourself and they just store the blob) and SSE-C (where AWS don't store your keys, you send them in every bucket API request). But even that latter method has other caveats: - What does the S3 service log? Who can access those logs? - Where does TLS for your S3 https request get terminated? Who can view the traffic? I'm assuming that this isn't just a regional issue, and that any AWS Support person globally could access buckets in any region. If so, then that's a big deal. If you're in Europe and your bank or healthcare provider is an AWS customer, how much trouble could you cause them (and by extension, AWS) right now? Furthermore, with the antiwork movement and backlash amongst employees for their treatment of warehouse workers, one cannot guarantee that an AWS worker wouldn't do something to hurt the company. Amazon need to head this of with a very thorough explanation of what happened and what was exposed directly and indirectly.
    - lmilcin 5y ago
      > Without having an AWS support person test each type and report back, one must assume that the only bulletproof s3 encryption methods are client-side It is normal practice to have a 3rd party access to your technical infrastructure (for example for purpose of support/maintenance). I was once contracted to maintain database for another company. You sign NDAs, you sign penalties, you sign your children to slavery and the right of first night with your wife. You know, standard business practice. But if you care enough that you would not have contracted 3rd party access to the data, client-side is the only solution assuming the client is under your sole control.
    - llarsson 5y ago
      > SSE-C (where AWS don't store your keys, you send them in every bucket API request) Since this is symmetrical encryption we're talking about, let's just be completely aware that the technical possibility to also store the encryption key definitely exists. It would violate the terms of service, of course. For those who don't know how SSE-C works, it's that you send both the unencrypted data and a key in a request. AWS will encrypt the data with the key, and store it encrypted. To get your data back, you supply the same key in your subsequent request. AWS will decrypt the data using the key, and send the unencrypted data back to you. During both those times when you gave AWS your key, you entirely trust that they will not also happen to store it for their own use.
deanCommie 5y ago
Reverted 22 hours ago: https://github.com/z0ph/MAMIP/commit/b6f696cebc7b9a4f71dd34c1da81287fa0230674#diff-6cf37356bf4a111ac75c84086acac5608bfc1390ba399181cc3ea6d6eb4afab3L1642 https://github.com/z0ph/MAMIP/commit/b6f696cebc7b9a4f71dd34c...
- StreamBright 5y ago
  Wow I was not aware of this existed. Thanks a million!
_3u10 5y ago
It’s their servers, they can do whatever they want with them. What are you going to do about it they have physical access, you have an API key. Suggestion, if you want to secure your data don’t put it on other peoples computers, and for fucks sake don’t store your crypto keys on someone else’s computer.
- deanCommie 5y ago
  A very shortsighted take. Sure, yes "they" can do whatever they want. But even in the world you are imagining where AWS is peeking at customer's data willy-nilly, I have to imagine you don't believe that every tech support representative should have default access to every AWS customer's storage data, do you? Even a dishonest unethical company that created backdoors for its employees would surely gate their backdoors. This change (a mistaken one that was rolled back immediately) would have given the keys through the front door to presumably thousand low-level employees. BTW, AWS spends a long time talking about how verifiably they do not have access to customer data. If you're interested in crypto (otherwise not sure why you are referencing it here), this kind of thing should be right up your alley: https://www.youtube.com/watch?v=4J8REvs7zaY https://www.youtube.com/watch?v=4J8REvs7zaY
  - _3u10 5y ago
    AWS has regions in China, they verifiably DO have access to your data. They also have regions in the US where they verifiably DO have access to your data. Both points of access are verifiable by their compliance with the law in those countries ensuring that the government can access that data. If you use their CA or EU locations it’s conceivable that they’ve developed separate software that actually protects your data but I would hazard a guess that they use the same backdoored software there once it has been sufficiently beta tested in us-east-1
- 878654Tom 5y ago
  That is a really false statement. This is why contracts, audits,... exists and they define what each party can and can't do. When in violation this could result in huge fines, loss of business,... You can also securely storage your data on other servers by using client-side encryption. Not every business/person has the means or knowledge to have their own datacenter.
ptcrash 5y ago
Okay, opinions up front: I don't think this is worthy of "declaring a security incident. Having some experience working behind the scenes, just because this policy was changes this way doesn't mean "All AWS Support personnel had unrequited access to your S3 objects." To me, this reads as Twitter inflammatory nonsense. Here's why: * KMS Encrypted objects would not be accessible because the support personnel would need permission policies that grant `kms:decrypot` permissions to encrypted objects. The only way this could wind up happening is if you are granting the AWS Support principal access in the KMS Key Policy. * Objects with a default-deny bucket policy could not have been circumvented with the support team's escalated privilege. So if you have a policy that looks something like this, that data was not exposed: { "Action": "Deny", "NotPrincipal": [...] } * Internal Checks. AWS has a lot of protections and checks in place to prevent their support personnel from accessing metadata about S3 objects. They don't have tools to fetch the actual objects unless your really high up the food-chain. Think like, people with a legal or security related reason to need to review data. Nevertheless, I'll share some nuggets of wisdom I've accrued over the years, in a hopes to save y'all some time: If you have an NDA with AWS, I'd recommend reaching out to your TAM and asking them about what the potential exposure was; and make sure to ask about the internal access control mechanisms. But everyone who's concerned and DIDN'T set up data access logging already: 1) Consider turning that on to trace potential disclosures in the future. 2) Open up a case with Premium Support under the CloudTrail, state that you have a security incident and you need to retrieve data events for the time of 2021-12-23 to 2021-12-22. If granted to you, save that sucker in S3 and query it for requests coming from `support.amazonaws.com` in Athena. [0] Hopefully this helps some of y'all. [0] https://aws.amazon.com/premiumsupport/knowledge-center/analyze-logs-athena/ https://aws.amazon.com/premiumsupport/knowledge-center/analy...
- ealexhudson 5y ago
  "Declaring an incident" means there's something to investigate, it didn't mean anything bad has happened: it's detection of a non-conformity. The output of the incident would look similar to what you wrote. Any time the wrong permissions are assigned and confidentiality is potentially breached, I think you have to have an incident. Arguably in some jurisdictions, it's a legal requirement to ensure you have a near miss not an actual breach.
rvz 5y ago
So its completely Amazon's storage then? Given that and it's not your computer, so it also not your storage (even it is encrypted)? If all of the above is true, is it time to panic? Perhaps maybe that why the cloud is somewhat of a scam anyways?
- midasuni 5y ago
  I think you’re a little harsh describing it as a scam, but yes you have to be aware of who you are trusting and what the risks and failure when using this type of service, including the security risks. Of course there are also risks with self hosting, depending on your competence and the service. I have a Vm on the cloud I use for running nightly nmaps against my network range, when that breaks it’s not the end of the world, if the VM provider hikes the price, or goes bust, I can migrate it in an hour by running a script, thetr no sensitive information on there, it makes perfect sense to run on a VM. Would I entrust my journalist’s files when they are investigating corruption in Amazon? No.
- exdsq 5y ago
  I do wonder if the push for cloud native development is just an inbound sales technique by Amazon/Microsoft/Google
twosugars 5y ago
When I worked in aws, this is primarily used to check for permissions of an object. I know how dumb customers can be, for the most part this is used to see why a customer cannot delete a bucket or object those sort of things. I don't remember having ability to see actual customers data only metadata is accessible. Edit: Based on what I know, I'm pretty sure support will not be able see any of the customers data.
- trzmiel4 5y ago
  > I know how dumb customers can be I find this insulting as a customer. Is AWS usually contemptuous of its customers? I don't think I've ever called my customer "dumb", and working as a consultant I've seen all kinds of interesting things. People make mistakes. They're always in a hurry. They may have a hard time understanding ambiguous, complex or incomplete documentation. The interface may be confusing and lead them to bad solutions. Come on, support is there to help. Take it easy, shall we?
  - twosugars 5y ago
    You sound like never worked in support area.
    - grumple 5y ago
      I've had many AWS support engineers (and higher engineers) look at things in our env and say "I've never seen that before" and have no clue what was happening. It's a two way street. Everybody can't know everything. And remember that many devs in the real world have much broader domains than AWS engineers - I have to know every nuance about 30 AWS services, as well as my own applications and my own domain. An AWS engineer would be limited to having a deep understanding of one or a few services, and has internal experts on individual services to reach out to when they don't have some information. But sometimes even AWS devs might not be aware of a little line in the Lambda docs like "Background processes or callbacks that were initiated by your Lambda function and did not complete when the function ended resume if Lambda reuses the execution environment. Make sure that any background processes or callbacks in your code are complete before the code exits." [1] There are gotchas like this with every service, and missing a single line within the novella of docs AWS provides for each service is not a significant failing. There are also issues and concerns that are completely undocumented and are only learned with experience. As a developer for a SaaS, I have to spend some time on support every day, including for devs who have refused to read our documentation for a particular service we provide (and the only one these devs use). It's frustrating, I know. You should assume that the developers who are your customers are unlikely to be stupid, and are instead just not informed about something or haven't read the docs (maybe they didn't know where to look, or like many, they are too busy to justify spending a day reading the docs for lambda). Best thing to do is direct them to the relevant parts of the documentation and do your best to help those people. 1. https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html https://docs.aws.amazon.com/lambda/latest/dg/runtimes-contex...
aborsy 5y ago
I have been looking into this lately for a company that wanted to host important data on AWS S3. I couldn’t find conclusive information in public domain. It’s hard to decipher the AWS Data Privacy policy: https://aws.amazon.com/compliance/data-privacy-faq/ https://aws.amazon.com/compliance/data-privacy-faq/ In section Who Owns Customer Content, it’s implied that AWS doesn’t access customers’ data: “As a customer, you maintain ownership of your content, and you select which AWS services can process, store, and host your content. We do not access or use your content for any purpose without your agreement. We never use customer content or derive information from it for marketing or advertising.” Here, the statement is we don’t access customer’s data without customer’s agreement. But customers do agree with ToS in which different statements might be included. In another place it’s mentioned, if governments request, AWS will comply. Considering that governments obviously request access from cloud providers, and obviously require them not to disclose the backdoor, I came to the conclusion that governments have unconditional access to data held in data centers of companies such as Amazon, Google and Microsoft. As for non-government access, there are claims that Amazon uses data or metadata to launch competing products. If I recall correctly, AWS may collect some data presumably to maintain and secure the platform and fight abuse. The counter argument is that, AWS will not risk a profitable business; but AWS is too big to be easily impacted and might act on good opportunities. It would be great if people working in AWS or similar platforms could chime in.
- Spooky23 5y ago
  When you think about risks it’s productive to understand what could happen vs what may. I was a stakeholder in an early cloud negotiation back in 2010-11, and one of the key issues was the support personnel. Ultimately, they have access to your data.
- luckylion 5y ago
  > AWS doesn’t access customers’ data Amazon's product people also don't look at third party seller statistics to decide which products to sell themselves. Until they got caught doing just that. To assume that they don't look at data feels naive. A German super market chain with online ambitions has a rule that nothing touching their pipeline can be hosted on AWS. Want to sell them SaaS? You can't run your nodes on AWS. I consider that to be reasonable.
grenran 5y ago
You do know that they can probably get access to your data through an internal backdoor if they really wanted to right?
- qwertyuiop_ 5y ago
  This ^
- motoboi 5y ago
  Yeah. I don’t think ls and has support to IAM yet.
- bspammer 5y ago
  No I don’t believe an average support agent can do this.
helsinkiandrew 5y ago
I'm not sure what the hoo hah is about. All that's changed is that this role is now visible in IAMS and any API calls by Amazon support tools will be logged in AWS CloudTrail - I don't think AWS have any more access now than they did before. It's obvious to anyone who has had problems with any AWS services (lambda functions that couldn't be deleted, non properly propagating/operational services) that support has access to them via their tools. https://docs.aws.amazon.com/awssupport/latest/user/using-service-linked-roles-sup.html https://docs.aws.amazon.com/awssupport/latest/user/using-ser...
- gonzo41 5y ago
  But can it override and explicit wildcard deny modification policy in a Bucket!
  - gunapologist99 5y ago
    Yes. As I wrote in https://news.ycombinator.com/item?id=29663566 https://news.ycombinator.com/item?id=29663566, service accounts are not constrained by customer bucket policies. In fact, not even SCP's are restricted by service-linked roles: "SCPs do not affect any service-linked role. Service-linked roles enable other AWS services to integrate with AWS Organizations and can't be restricted by SCPs." https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html https://docs.aws.amazon.com/organizations/latest/userguide/o...
- skywhopper 5y ago
  The policy in question had “s3:GetObject” permission to “*” added for a few hours. And CloudTrail logging doesn’t capture GetObject API requests by default. This role is specifically for metadata only access. Yes, service teams have behind-the-scenes escalation tools for bugs on the backend, but the more numerous front-line support staff should only be able to view metadata. The GetObject permission would have let them view actual potentially sensitive data.
  - tills13 5y ago
    You really shouldn't be storing "potentially sensitive data" in a bucket unencrypted, though.
L0g4n 5y ago
Such concerns specifically led to my decision of only uploading sensitive data to S3 with client side encryption. Since the aws cli tool only supports server side encryption with keys stored on amazon servers (where the non-default managed keys cost like 1 USD per month), I decided to simply symmetrically encrypt the backup of my syncthing data volume with AES256 using gnupg and only then pushing it to the S3 bucket.
- geek_at 5y ago
  this should be the default user behaviour for any cloud storage. Don't put unencrypted (company) data on a cloud infrastructure you don't have full control over. Also reminds me of the (hyped?) "outrage" when a former facebook developer stated that they used to have a "default password" that allowed fb devs to log into every account and the media were like "omg they could have logged in and seen your photos". I mean... yeah they're the developers they could always do that even without the password
  - krisoft 5y ago
    > yeah they're the developers they could always do that even without the password Not really. Obviously facebook the company can always access your data. Weather or not an individual developer can do the same, which developers can do it, how they can do it, and under what level of supervision this would be is a design choice. It is possible to design a system with very high level of security and ones with none too. As with any design considerations it has trade-offs. A super secure system might introduce dev and operational frictions which the company might deem unnacceptable. But even with that consideration the question is a lot more complicated than a simple “yeah they’re the developers”.
    - oneplane 5y ago
      Sadly, "yeah, they are the developers" applies far more often than any other scenario. Unless a business is heavily regulated and checked for compliance, the burden and friction introduced by developer access controls to the data of the software they write is anecdotally not seen as a positive investment in any company I've seen the internals of.
harshaw 5y ago
I guess taking some risks as I am not really authorized to use social media, but at AWS there is a sauron like focus on not letting internal engineers view customer data. On my service we don't even have tools to do it and the security controls are very tight and getting tighter all the time. It's a big fucking deal, if not the biggest fucking deal, besides KTLO. Posting to get ahead of some of the comment here on security posture.
- jonahbenton 5y ago
  I thought this was true at the engineering level, but "authorized" cross customer data sharing- where there is some debate at what's customer data vs what's platform data- at the business level seems...rampant? Just curious for more perspective on posture.
- dmead 5y ago
  KTLO?
  - aaronmdjones 5y ago
    Keeping the Lights On.
  - go_blue_13 5y ago
    LMGTFY
    - dmead 5y ago
      I did, and it's a radio station.
      - thathndude 5y ago
        You’re listening to KTLO, all the hits, all day . . . Rolls off the tongue
      - smolder 5y ago
        The acronym was like 2nd or 3rd result on ddg after the radio station.
ryanschneider 5y ago
I’m honestly surprised no one has mentioned log4j in this thread yet. What if this change was part of an outside attack?
- staticassertion 5y ago
  There's no reason to believe that to be the case.
- gunapologist99 5y ago
  That's an excellent point, especially since AWS has a big Java presence.
joelbondurant0 5y ago
deleted 5y ago
[deleted]
seandavidfisher 5y ago
Official AWS response: https://aws.amazon.com/security/security-bulletins/AWS-2021-007/ https://aws.amazon.com/security/security-bulletins/AWS-2021-...

450 points by zdw 5y ago

rronalddas 5y ago

KMS Encrypted objects shouldn't be affected though

cateof 5y ago
Objects encrypted with S3-managed encryption keys (SSE-S3) are affected, as these keys are set up with a non-configurable resource policy granting the S3 service decryption permissions.
hericium 5y ago
Aren't KMS keys created by Amazon?
- RKearney 5y ago
  Yes but this role did not add the necessary privileges for it to use customer KMS keys. You can’t get an S3 object that’s encrypted with a KMS key if you don’t also have permission to decrypt with that key. Of course Amazon could just give themselves access to decrypt with your KMS keys too, but that didn’t happen here.
- raffraffraff 5y ago
  There are three types of S3 server side encryption: - SSE-KMS - SSE-S3 - SSE-C Without having an AWS support person test each type and report back, one must assume that the only bulletproof s3 encryption methods are client-side (where you handle encryption and decryption yourself and they just store the blob) and SSE-C (where AWS don't store your keys, you send them in every bucket API request). But even that latter method has other caveats: - What does the S3 service log? Who can access those logs? - Where does TLS for your S3 https request get terminated? Who can view the traffic? I'm assuming that this isn't just a regional issue, and that any AWS Support person globally could access buckets in any region. If so, then that's a big deal. If you're in Europe and your bank or healthcare provider is an AWS customer, how much trouble could you cause them (and by extension, AWS) right now? Furthermore, with the antiwork movement and backlash amongst employees for their treatment of warehouse workers, one cannot guarantee that an AWS worker wouldn't do something to hurt the company. Amazon need to head this of with a very thorough explanation of what happened and what was exposed directly and indirectly.
  - lmilcin 5y ago
    > Without having an AWS support person test each type and report back, one must assume that the only bulletproof s3 encryption methods are client-side It is normal practice to have a 3rd party access to your technical infrastructure (for example for purpose of support/maintenance). I was once contracted to maintain database for another company. You sign NDAs, you sign penalties, you sign your children to slavery and the right of first night with your wife. You know, standard business practice. But if you care enough that you would not have contracted 3rd party access to the data, client-side is the only solution assuming the client is under your sole control.
  - llarsson 5y ago
    > SSE-C (where AWS don't store your keys, you send them in every bucket API request) Since this is symmetrical encryption we're talking about, let's just be completely aware that the technical possibility to also store the encryption key definitely exists. It would violate the terms of service, of course. For those who don't know how SSE-C works, it's that you send both the unencrypted data and a key in a request. AWS will encrypt the data with the key, and store it encrypted. To get your data back, you supply the same key in your subsequent request. AWS will decrypt the data using the key, and send the unencrypted data back to you. During both those times when you gave AWS your key, you entirely trust that they will not also happen to store it for their own use.

deanCommie 5y ago

Reverted 22 hours ago: https://github.com/z0ph/MAMIP/commit/b6f696cebc7b9a4f71dd34c1da81287fa0230674#diff-6cf37356bf4a111ac75c84086acac5608bfc1390ba399181cc3ea6d6eb4afab3L1642 https://github.com/z0ph/MAMIP/commit/b6f696cebc7b9a4f71dd34c...

StreamBright 5y ago
Wow I was not aware of this existed. Thanks a million!

_3u10 5y ago

It’s their servers, they can do whatever they want with them. What are you going to do about it they have physical access, you have an API key. Suggestion, if you want to secure your data don’t put it on other peoples computers, and for fucks sake don’t store your crypto keys on someone else’s computer.

deanCommie 5y ago
A very shortsighted take. Sure, yes "they" can do whatever they want. But even in the world you are imagining where AWS is peeking at customer's data willy-nilly, I have to imagine you don't believe that every tech support representative should have default access to every AWS customer's storage data, do you? Even a dishonest unethical company that created backdoors for its employees would surely gate their backdoors. This change (a mistaken one that was rolled back immediately) would have given the keys through the front door to presumably thousand low-level employees. BTW, AWS spends a long time talking about how verifiably they do not have access to customer data. If you're interested in crypto (otherwise not sure why you are referencing it here), this kind of thing should be right up your alley: https://www.youtube.com/watch?v=4J8REvs7zaY https://www.youtube.com/watch?v=4J8REvs7zaY
- _3u10 5y ago
  AWS has regions in China, they verifiably DO have access to your data. They also have regions in the US where they verifiably DO have access to your data. Both points of access are verifiable by their compliance with the law in those countries ensuring that the government can access that data. If you use their CA or EU locations it’s conceivable that they’ve developed separate software that actually protects your data but I would hazard a guess that they use the same backdoored software there once it has been sufficiently beta tested in us-east-1
878654Tom 5y ago
That is a really false statement. This is why contracts, audits,... exists and they define what each party can and can't do. When in violation this could result in huge fines, loss of business,... You can also securely storage your data on other servers by using client-side encryption. Not every business/person has the means or knowledge to have their own datacenter.

ptcrash 5y ago

Okay, opinions up front: I don't think this is worthy of "declaring a security incident. Having some experience working behind the scenes, just because this policy was changes this way doesn't mean "All AWS Support personnel had unrequited access to your S3 objects." To me, this reads as Twitter inflammatory nonsense. Here's why: * KMS Encrypted objects would not be accessible because the support personnel would need permission policies that grant `kms:decrypot` permissions to encrypted objects. The only way this could wind up happening is if you are granting the AWS Support principal access in the KMS Key Policy. * Objects with a default-deny bucket policy could not have been circumvented with the support team's escalated privilege. So if you have a policy that looks something like this, that data was not exposed: { "Action": "Deny", "NotPrincipal": [...] } * Internal Checks. AWS has a lot of protections and checks in place to prevent their support personnel from accessing metadata about S3 objects. They don't have tools to fetch the actual objects unless your really high up the food-chain. Think like, people with a legal or security related reason to need to review data. Nevertheless, I'll share some nuggets of wisdom I've accrued over the years, in a hopes to save y'all some time: If you have an NDA with AWS, I'd recommend reaching out to your TAM and asking them about what the potential exposure was; and make sure to ask about the internal access control mechanisms. But everyone who's concerned and DIDN'T set up data access logging already: 1) Consider turning that on to trace potential disclosures in the future. 2) Open up a case with Premium Support under the CloudTrail, state that you have a security incident and you need to retrieve data events for the time of 2021-12-23 to 2021-12-22. If granted to you, save that sucker in S3 and query it for requests coming from `support.amazonaws.com` in Athena. [0] Hopefully this helps some of y'all. [0] https://aws.amazon.com/premiumsupport/knowledge-center/analyze-logs-athena/ https://aws.amazon.com/premiumsupport/knowledge-center/analy...

ealexhudson 5y ago
"Declaring an incident" means there's something to investigate, it didn't mean anything bad has happened: it's detection of a non-conformity. The output of the incident would look similar to what you wrote. Any time the wrong permissions are assigned and confidentiality is potentially breached, I think you have to have an incident. Arguably in some jurisdictions, it's a legal requirement to ensure you have a near miss not an actual breach.

rvz 5y ago

So its completely Amazon's storage then? Given that and it's not your computer, so it also not your storage (even it is encrypted)? If all of the above is true, is it time to panic? Perhaps maybe that why the cloud is somewhat of a scam anyways?

midasuni 5y ago
I think you’re a little harsh describing it as a scam, but yes you have to be aware of who you are trusting and what the risks and failure when using this type of service, including the security risks. Of course there are also risks with self hosting, depending on your competence and the service. I have a Vm on the cloud I use for running nightly nmaps against my network range, when that breaks it’s not the end of the world, if the VM provider hikes the price, or goes bust, I can migrate it in an hour by running a script, thetr no sensitive information on there, it makes perfect sense to run on a VM. Would I entrust my journalist’s files when they are investigating corruption in Amazon? No.
exdsq 5y ago
I do wonder if the push for cloud native development is just an inbound sales technique by Amazon/Microsoft/Google

twosugars 5y ago

When I worked in aws, this is primarily used to check for permissions of an object. I know how dumb customers can be, for the most part this is used to see why a customer cannot delete a bucket or object those sort of things. I don't remember having ability to see actual customers data only metadata is accessible. Edit: Based on what I know, I'm pretty sure support will not be able see any of the customers data.

trzmiel4 5y ago
> I know how dumb customers can be I find this insulting as a customer. Is AWS usually contemptuous of its customers? I don't think I've ever called my customer "dumb", and working as a consultant I've seen all kinds of interesting things. People make mistakes. They're always in a hurry. They may have a hard time understanding ambiguous, complex or incomplete documentation. The interface may be confusing and lead them to bad solutions. Come on, support is there to help. Take it easy, shall we?
- twosugars 5y ago
  You sound like never worked in support area.
  - grumple 5y ago
    I've had many AWS support engineers (and higher engineers) look at things in our env and say "I've never seen that before" and have no clue what was happening. It's a two way street. Everybody can't know everything. And remember that many devs in the real world have much broader domains than AWS engineers - I have to know every nuance about 30 AWS services, as well as my own applications and my own domain. An AWS engineer would be limited to having a deep understanding of one or a few services, and has internal experts on individual services to reach out to when they don't have some information. But sometimes even AWS devs might not be aware of a little line in the Lambda docs like "Background processes or callbacks that were initiated by your Lambda function and did not complete when the function ended resume if Lambda reuses the execution environment. Make sure that any background processes or callbacks in your code are complete before the code exits." [1] There are gotchas like this with every service, and missing a single line within the novella of docs AWS provides for each service is not a significant failing. There are also issues and concerns that are completely undocumented and are only learned with experience. As a developer for a SaaS, I have to spend some time on support every day, including for devs who have refused to read our documentation for a particular service we provide (and the only one these devs use). It's frustrating, I know. You should assume that the developers who are your customers are unlikely to be stupid, and are instead just not informed about something or haven't read the docs (maybe they didn't know where to look, or like many, they are too busy to justify spending a day reading the docs for lambda). Best thing to do is direct them to the relevant parts of the documentation and do your best to help those people. 1. https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html https://docs.aws.amazon.com/lambda/latest/dg/runtimes-contex...

aborsy 5y ago

I have been looking into this lately for a company that wanted to host important data on AWS S3. I couldn’t find conclusive information in public domain. It’s hard to decipher the AWS Data Privacy policy: https://aws.amazon.com/compliance/data-privacy-faq/ https://aws.amazon.com/compliance/data-privacy-faq/ In section Who Owns Customer Content, it’s implied that AWS doesn’t access customers’ data: “As a customer, you maintain ownership of your content, and you select which AWS services can process, store, and host your content. We do not access or use your content for any purpose without your agreement. We never use customer content or derive information from it for marketing or advertising.” Here, the statement is we don’t access customer’s data without customer’s agreement. But customers do agree with ToS in which different statements might be included. In another place it’s mentioned, if governments request, AWS will comply. Considering that governments obviously request access from cloud providers, and obviously require them not to disclose the backdoor, I came to the conclusion that governments have unconditional access to data held in data centers of companies such as Amazon, Google and Microsoft. As for non-government access, there are claims that Amazon uses data or metadata to launch competing products. If I recall correctly, AWS may collect some data presumably to maintain and secure the platform and fight abuse. The counter argument is that, AWS will not risk a profitable business; but AWS is too big to be easily impacted and might act on good opportunities. It would be great if people working in AWS or similar platforms could chime in.

Spooky23 5y ago
When you think about risks it’s productive to understand what could happen vs what may. I was a stakeholder in an early cloud negotiation back in 2010-11, and one of the key issues was the support personnel. Ultimately, they have access to your data.
luckylion 5y ago
> AWS doesn’t access customers’ data Amazon's product people also don't look at third party seller statistics to decide which products to sell themselves. Until they got caught doing just that. To assume that they don't look at data feels naive. A German super market chain with online ambitions has a rule that nothing touching their pipeline can be hosted on AWS. Want to sell them SaaS? You can't run your nodes on AWS. I consider that to be reasonable.

grenran 5y ago

You do know that they can probably get access to your data through an internal backdoor if they really wanted to right?

qwertyuiop_ 5y ago
This ^
motoboi 5y ago
Yeah. I don’t think ls and has support to IAM yet.
bspammer 5y ago
No I don’t believe an average support agent can do this.

helsinkiandrew 5y ago

I'm not sure what the hoo hah is about. All that's changed is that this role is now visible in IAMS and any API calls by Amazon support tools will be logged in AWS CloudTrail - I don't think AWS have any more access now than they did before. It's obvious to anyone who has had problems with any AWS services (lambda functions that couldn't be deleted, non properly propagating/operational services) that support has access to them via their tools. https://docs.aws.amazon.com/awssupport/latest/user/using-service-linked-roles-sup.html https://docs.aws.amazon.com/awssupport/latest/user/using-ser...

gonzo41 5y ago
But can it override and explicit wildcard deny modification policy in a Bucket!
- gunapologist99 5y ago
  Yes. As I wrote in https://news.ycombinator.com/item?id=29663566 https://news.ycombinator.com/item?id=29663566, service accounts are not constrained by customer bucket policies. In fact, not even SCP's are restricted by service-linked roles: "SCPs do not affect any service-linked role. Service-linked roles enable other AWS services to integrate with AWS Organizations and can't be restricted by SCPs." https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html https://docs.aws.amazon.com/organizations/latest/userguide/o...
skywhopper 5y ago
The policy in question had “s3:GetObject” permission to “*” added for a few hours. And CloudTrail logging doesn’t capture GetObject API requests by default. This role is specifically for metadata only access. Yes, service teams have behind-the-scenes escalation tools for bugs on the backend, but the more numerous front-line support staff should only be able to view metadata. The GetObject permission would have let them view actual potentially sensitive data.
- tills13 5y ago
  You really shouldn't be storing "potentially sensitive data" in a bucket unencrypted, though.

L0g4n 5y ago

Such concerns specifically led to my decision of only uploading sensitive data to S3 with client side encryption. Since the aws cli tool only supports server side encryption with keys stored on amazon servers (where the non-default managed keys cost like 1 USD per month), I decided to simply symmetrically encrypt the backup of my syncthing data volume with AES256 using gnupg and only then pushing it to the S3 bucket.

geek_at 5y ago
this should be the default user behaviour for any cloud storage. Don't put unencrypted (company) data on a cloud infrastructure you don't have full control over. Also reminds me of the (hyped?) "outrage" when a former facebook developer stated that they used to have a "default password" that allowed fb devs to log into every account and the media were like "omg they could have logged in and seen your photos". I mean... yeah they're the developers they could always do that even without the password
- krisoft 5y ago
  > yeah they're the developers they could always do that even without the password Not really. Obviously facebook the company can always access your data. Weather or not an individual developer can do the same, which developers can do it, how they can do it, and under what level of supervision this would be is a design choice. It is possible to design a system with very high level of security and ones with none too. As with any design considerations it has trade-offs. A super secure system might introduce dev and operational frictions which the company might deem unnacceptable. But even with that consideration the question is a lot more complicated than a simple “yeah they’re the developers”.
  - oneplane 5y ago
    Sadly, "yeah, they are the developers" applies far more often than any other scenario. Unless a business is heavily regulated and checked for compliance, the burden and friction introduced by developer access controls to the data of the software they write is anecdotally not seen as a positive investment in any company I've seen the internals of.

harshaw 5y ago

I guess taking some risks as I am not really authorized to use social media, but at AWS there is a sauron like focus on not letting internal engineers view customer data. On my service we don't even have tools to do it and the security controls are very tight and getting tighter all the time. It's a big fucking deal, if not the biggest fucking deal, besides KTLO. Posting to get ahead of some of the comment here on security posture.

jonahbenton 5y ago
I thought this was true at the engineering level, but "authorized" cross customer data sharing- where there is some debate at what's customer data vs what's platform data- at the business level seems...rampant? Just curious for more perspective on posture.
dmead 5y ago
KTLO?
- aaronmdjones 5y ago
  Keeping the Lights On.
- go_blue_13 5y ago
  LMGTFY
  - dmead 5y ago
    I did, and it's a radio station.
    - thathndude 5y ago
      You’re listening to KTLO, all the hits, all day . . . Rolls off the tongue
    - smolder 5y ago
      The acronym was like 2nd or 3rd result on ddg after the radio station.

ryanschneider 5y ago

I’m honestly surprised no one has mentioned log4j in this thread yet. What if this change was part of an outside attack?

staticassertion 5y ago
There's no reason to believe that to be the case.
gunapologist99 5y ago
That's an excellent point, especially since AWS has a big Java presence.

joelbondurant0 5y ago

deleted 5y ago

[deleted]

seandavidfisher 5y ago

Official AWS response: https://aws.amazon.com/security/security-bulletins/AWS-2021-007/ https://aws.amazon.com/security/security-bulletins/AWS-2021-...