17 ms·

What types of AWS data would be trawled? Are we talking about data inside S3 buckets, database schemas, particular architecure styles, the fact that a product i

What types of AWS data would be trawled? Are we talking about data inside S3 buckets, database schemas, particular architecure styles, the fact that a product is consuming {x, y, z} amounts of cloud resources, or simply "spending $m / year" in gross?

ShroudedNight 6y ago
Given how granular AWS billing data is, I would expect the odds to be fairly good that it alone is sufficient to make a good analysis for which third-party offerings are compelling markets. Then AWS takes their execution advantage, along with things like the lower friction that arises from first-party integration with IAM and billing, as well as not having to pay retail for the cloud resources, and it becomes very difficult to retain a moat unless you have a paradigm or perspective that is both critical to succeeding and is also incompatible with AWS culture.
- QuinnyPig 6y ago
  You’re correct. It’s disturbingly detailed as far as what it reveals about architecture.
whoisjuan 6y ago
I can confidently tell you that Amazon's employees cannot see customers data inside S3 buckets or EC2 instances. They are extremely serious about that stuff since they know that will erode their customer's confidence. But there's probably other superficial business data that's helpful to evaluate that.
- KorfmannArno 6y ago
  1. Did you work on a team at Amazon in the likes of what user throwaway_aws mentioned? 2. What measures that you know of is Amazon implementing to make sure no employees across all teams are having access to said resources?
  - whoisjuan 6y ago
    As I said below this is something that they will talk a about like every freaking day. They talk about customer’s data as the most important thing to take care of. Basically is preferable to get a bullet in the head than to ever reveal or tamper with customer’s data. I cannot answer your question about who has access or not but I’m telling you what’s the culture when it comes to customer’s data. At the end of the day I was just another IC doing menial work so probably not a good reference, but that was my experience
- kaesar14 6y ago
  Amazon is a massive company. How can you know this with confidence? Are you in the C-Suite?
  - whoisjuan 6y ago
    It’s the thing they tell you the most when you work there. Like in a a obnoxious way. Most infosec training is about that. If someone has access to customer’s data for their work they have to do a bunch of extra training and do other stuff. Potentially sign some things and there’s probably a different way to authenticate. I really don’t know because I never had to do that and nobody I knew had that type of access but I heard when you do you have to put with more things.
    - kaesar14 6y ago
      But then what about other commenters saying that this is exactly what their sectors of the company do? Do you think it's impossible that a massive company like Amazon that controls an ungodly amount of the Internet would break those rules? Especially when the government of their home country hasn't pursued an antitrust case in God knows how long
      - whoisjuan 6y ago
        Yeah I don’t know. It’s possible that there’s some evil stuff happening. I’m just relating my experience as a pawn employee. They parrot this incessantly.
      - notatoad 6y ago
        >But then what about other commenters saying that this is exactly what their sectors of the company do? i don't see anybody claiming that amazon is harvesting data from inside their customer's infrastructure. amazon has a lot of data that's "amazon's data" that would tell them about businesses that are operating on AWS that might be ripe for competition. For example, they know what your AWS bill is, and how it's been trending. If you pay a huge bandwidth bill and it goes up 50% each month, they know you've got a business model that's working and that they can undercut you on one of your big expenses.
- TimSchumann 6y ago
  > I can confidently tell you that Amazon's employees cannot see customers data inside S3 buckets or EC2 instances. From a technical standpoint, that statement is false. Every employee might not have the credentials to, but for AWS to function as it does, SOMEONE inside the company has to have those credentials. If you change 'cannot' to 'don't', well then we've just gotta take you at your word, which is where we started anyway.
  - whoisjuan 6y ago
    Yeah. True. I guess what I meant is that just a handful of employees have access to that and they need to have legitimate reasons.
  - ibejoeb 6y ago
    The critical factor here is whether there are controls in place to prevent it. Sure, somebody probably could, but what to what lengths must that person go to do it, and what happens when it is discovered? Most things are not technically impossible, after all.
  - count 6y ago
    Except they get audited by 3rd parties on statements like that, and have controls tested. It's not like they're just ... digital ocean or somebody.
    - deleted 6y ago
      [deleted]
    - jasonjayr 6y ago
      Do you have evidence of this claim re DO? I worked with a DO on an technical issue, and they were steadfastly against me granting them temporary access to our servers even though it would have made the issue easier to diagnose. Cloud provider that verifiably get caught doing this will quickly lose the trust of all their large customers
      - webo 6y ago
- kerng 6y ago
  This is incorrect, at least from a logical POV and why it's hard to trust what cloud vendors say. A statement like this is either naive (most likely) or actively attempting to mislead. Technically, its absolutely possible. Most likely you'll just need a support ticket or bug, and then you can troll around as engineer. Also, security teams also usually have access to stuff when things get interesting. Better to say that access is strictly on a case by case basis and monitored thoroughly. Ideally customer is notified each time it happens - that would be cool, but likely technically not possible since data ends up in so many systems (like logs, SIEM, telemetry, debug files, backups, data scientist desktops,....)
  - whoisjuan 6y ago
    I already corrected my statement in another reply. You're right. I said probably only a handful of people can access customer data to do their job. I personally never met one. The goal of my comment was to illustrate that in my experience handling customer data there was a big deal. It's not like something you can casually query to see if a particular customer has a good business or not.
  - donavanm 6y ago
    > Ideally customer is notified each time it happens - that would be cool, but likely technically not possible You're underestimating the investments that AWS (and Amazon at large) make in to security, confidentiality, and auditing. You're also missing a fundamental implication of building AWS on AWS primitives. As a relevant example there is only one AWS IAM and one CloudTrail. It's a core tenant of AWS IAM to put that control and root of trust in to the customers control. That means when developer support is helping with your ticket they do so via your accounts AWSServiceRoleForSupport role. That means you can control whether that role exists, which principals can assume it, the capabilities it has, and you can see those same API calls in your CloudTrail logs. Although it would make support difficult you're welcome to delete that service linked role and prevent support.amazonaws.com from assuming said role in your account. https://docs.aws.amazon.com/awssupport/latest/user/using-service-linked-roles-sup.html https://docs.aws.amazon.com/awssupport/latest/user/using-ser...
    - 6y ago
- luizfelberti 6y ago
  I'm sorry but what you just said is patently false: https://www.bloomberg.com/news/articles/2019-07-29/capital-one-data-systems-breached-by-seattle-woman-u-s-says https://www.bloomberg.com/news/articles/2019-07-29/capital-o... Quote: Capital One Financial Corp. said data from about 100 million people in the U.S. was illegally accessed after prosecutors accused a Seattle woman identified by Amazon.com Inc. as one of its former cloud service employees of breaking into the bank’s server. While the complaint doesn’t identify the cloud provider that stored the allegedly stolen data, the charging papers mention information stored in S3, a reference to Simple Storage Service, Amazon Web Services’ popular data storage software.
  - jburgess777 6y ago
    My reading of this is that the ex-employee used the knowledge about EC2 instance credentials being accessible as a path to gain unauthorized access to data. In theory anyone could have exploited this vulnerability even if they had never worked for Amazon. They never say that Amazon employees had privileged credentaials that would give them unauthorized access to customer data. AWS customers that want to avoid this vulnerability should disable IMDSv1 as per https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/ https://aws.amazon.com/blogs/security/defense-in-depth-open-...
    - donavanm 6y ago
      There was zero inside knowledge and they were an ex employee at all times relevant to the incident. The EC2 instance credentials via the metadata url is public documented functionality. Its how things like the SDK “just work.” The S3 bucket policy, instance creds, and (inferred) overly permissive IAM policy is all public documented functionality. This looks like a simple case of an initial intrusion being escalated via permissive configuration and controls. There would be no story if the suspect had not been employed by AWS in the past. Disclaimer: Im a Principal jn AWS but have no direct or inside knowledge of this incident. Everything I know or have stated here is public record (eg the indictment) or public AWS docs.
  - didibus 6y ago
bg24 6y ago
Can speak for AWS. Only the later. Basically the usage information for cloud resources. This constitutes the foundation for billing. BTW, this is be true for any cloud, any SAAS. There is no way an employee can look into customer data. There's enough trail inside AWS to prove that without any doubt.
- KorfmannArno 6y ago
  What are the measures being implemented to ensure that no employee can look into customer's data?
  - yandie 6y ago
    I used to work for AWS and had to deep dive into IAM to build a feature. Basically Everytime you touch AWS your session is tagged with your credentials and has a unique ID. So everything downstream you touch has your session ID associated with it. Now say somebody from Redshift wants to access the customer's data. They will then need to access to the encryption key in KMS. The trail will be there since KMS lives in the customer's account (you can audit your own access). And for production services, human actors cannot access these keys - only production credentials can. An engineer who can log into a prod host in theory can grab the temporary credentials there but it expires in 15 minutes so your trail will be rather visible. Also access to prod host has a high bar - only senior people can do it. Now in theory somebody can coordinate with a malicious user in KMS team - but the bar is high. Also the actual master key never leaves the premise for KMS so your attack surface is very limited. Of course there are some core teams like IAM and KMS where if they become vulnerable the whole thing falls apart. But that's a big stretch for those systems since they are the core to the business.
    - jacquesm 6y ago
      This is about as bad a revelation as the original one. So the encryption key is fair game without explicit customer approval?
      - 6y ago
redredrobot 6y ago
I worked in an area where it is really hard to figure out exactly what workloads were being run and where it would have been extremely useful to know even basic things like CPU utilization patterns, network throughput patterns, etc for a specific customer. We had access to absolutely none of that information. We flew blind, relying entirely on the fact that we gave our customers enough hand-holding support that they would willingly volunteer information about their workloads so we could help them optimize it/save money. No one even attempted to get more detailed customer information AFAIK because it would have been extremely against company culture. That isn't Earning Trust or having Customer Obsession. The idea of reading data in someone's S3 bucket or inspecting what is happening inside of someone's EC2 instance in any way was unthinkable. Amazon is huge and imperfect, but from what I saw AWS takes data privacy extremely seriously.
kapilvt 6y ago
aggregated api usage stats, api client headers is often enough to identify competitor products and their traction, and is non-sensitive, coupled with account id to customers.
- anonymousDan 6y ago
  Do you have to use AWS to sell on Amazon?
  - kapilvt 6y ago
    no

ShroudedNight 6y ago

Given how granular AWS billing data is, I would expect the odds to be fairly good that it alone is sufficient to make a good analysis for which third-party offerings are compelling markets. Then AWS takes their execution advantage, along with things like the lower friction that arises from first-party integration with IAM and billing, as well as not having to pay retail for the cloud resources, and it becomes very difficult to retain a moat unless you have a paradigm or perspective that is both critical to succeeding and is also incompatible with AWS culture.

QuinnyPig 6y ago
You’re correct. It’s disturbingly detailed as far as what it reveals about architecture.

whoisjuan 6y ago

I can confidently tell you that Amazon's employees cannot see customers data inside S3 buckets or EC2 instances. They are extremely serious about that stuff since they know that will erode their customer's confidence. But there's probably other superficial business data that's helpful to evaluate that.

KorfmannArno 6y ago
1. Did you work on a team at Amazon in the likes of what user throwaway_aws mentioned? 2. What measures that you know of is Amazon implementing to make sure no employees across all teams are having access to said resources?
- whoisjuan 6y ago
  As I said below this is something that they will talk a about like every freaking day. They talk about customer’s data as the most important thing to take care of. Basically is preferable to get a bullet in the head than to ever reveal or tamper with customer’s data. I cannot answer your question about who has access or not but I’m telling you what’s the culture when it comes to customer’s data. At the end of the day I was just another IC doing menial work so probably not a good reference, but that was my experience
kaesar14 6y ago
Amazon is a massive company. How can you know this with confidence? Are you in the C-Suite?
- whoisjuan 6y ago
  It’s the thing they tell you the most when you work there. Like in a a obnoxious way. Most infosec training is about that. If someone has access to customer’s data for their work they have to do a bunch of extra training and do other stuff. Potentially sign some things and there’s probably a different way to authenticate. I really don’t know because I never had to do that and nobody I knew had that type of access but I heard when you do you have to put with more things.
  - kaesar14 6y ago
    But then what about other commenters saying that this is exactly what their sectors of the company do? Do you think it's impossible that a massive company like Amazon that controls an ungodly amount of the Internet would break those rules? Especially when the government of their home country hasn't pursued an antitrust case in God knows how long
    - whoisjuan 6y ago
      Yeah I don’t know. It’s possible that there’s some evil stuff happening. I’m just relating my experience as a pawn employee. They parrot this incessantly.
    - notatoad 6y ago
      >But then what about other commenters saying that this is exactly what their sectors of the company do? i don't see anybody claiming that amazon is harvesting data from inside their customer's infrastructure. amazon has a lot of data that's "amazon's data" that would tell them about businesses that are operating on AWS that might be ripe for competition. For example, they know what your AWS bill is, and how it's been trending. If you pay a huge bandwidth bill and it goes up 50% each month, they know you've got a business model that's working and that they can undercut you on one of your big expenses.
TimSchumann 6y ago
> I can confidently tell you that Amazon's employees cannot see customers data inside S3 buckets or EC2 instances. From a technical standpoint, that statement is false. Every employee might not have the credentials to, but for AWS to function as it does, SOMEONE inside the company has to have those credentials. If you change 'cannot' to 'don't', well then we've just gotta take you at your word, which is where we started anyway.
- whoisjuan 6y ago
  Yeah. True. I guess what I meant is that just a handful of employees have access to that and they need to have legitimate reasons.
- ibejoeb 6y ago
  The critical factor here is whether there are controls in place to prevent it. Sure, somebody probably could, but what to what lengths must that person go to do it, and what happens when it is discovered? Most things are not technically impossible, after all.
- count 6y ago
  Except they get audited by 3rd parties on statements like that, and have controls tested. It's not like they're just ... digital ocean or somebody.
  - deleted 6y ago
    [deleted]
  - jasonjayr 6y ago
    Do you have evidence of this claim re DO? I worked with a DO on an technical issue, and they were steadfastly against me granting them temporary access to our servers even though it would have made the issue easier to diagnose. Cloud provider that verifiably get caught doing this will quickly lose the trust of all their large customers
    - webo 6y ago
kerng 6y ago
This is incorrect, at least from a logical POV and why it's hard to trust what cloud vendors say. A statement like this is either naive (most likely) or actively attempting to mislead. Technically, its absolutely possible. Most likely you'll just need a support ticket or bug, and then you can troll around as engineer. Also, security teams also usually have access to stuff when things get interesting. Better to say that access is strictly on a case by case basis and monitored thoroughly. Ideally customer is notified each time it happens - that would be cool, but likely technically not possible since data ends up in so many systems (like logs, SIEM, telemetry, debug files, backups, data scientist desktops,....)
- whoisjuan 6y ago
  I already corrected my statement in another reply. You're right. I said probably only a handful of people can access customer data to do their job. I personally never met one. The goal of my comment was to illustrate that in my experience handling customer data there was a big deal. It's not like something you can casually query to see if a particular customer has a good business or not.
- donavanm 6y ago
  > Ideally customer is notified each time it happens - that would be cool, but likely technically not possible You're underestimating the investments that AWS (and Amazon at large) make in to security, confidentiality, and auditing. You're also missing a fundamental implication of building AWS on AWS primitives. As a relevant example there is only one AWS IAM and one CloudTrail. It's a core tenant of AWS IAM to put that control and root of trust in to the customers control. That means when developer support is helping with your ticket they do so via your accounts AWSServiceRoleForSupport role. That means you can control whether that role exists, which principals can assume it, the capabilities it has, and you can see those same API calls in your CloudTrail logs. Although it would make support difficult you're welcome to delete that service linked role and prevent support.amazonaws.com from assuming said role in your account. https://docs.aws.amazon.com/awssupport/latest/user/using-service-linked-roles-sup.html https://docs.aws.amazon.com/awssupport/latest/user/using-ser...
  - 6y ago
luizfelberti 6y ago
I'm sorry but what you just said is patently false: https://www.bloomberg.com/news/articles/2019-07-29/capital-one-data-systems-breached-by-seattle-woman-u-s-says https://www.bloomberg.com/news/articles/2019-07-29/capital-o... Quote: Capital One Financial Corp. said data from about 100 million people in the U.S. was illegally accessed after prosecutors accused a Seattle woman identified by Amazon.com Inc. as one of its former cloud service employees of breaking into the bank’s server. While the complaint doesn’t identify the cloud provider that stored the allegedly stolen data, the charging papers mention information stored in S3, a reference to Simple Storage Service, Amazon Web Services’ popular data storage software.
- jburgess777 6y ago
  My reading of this is that the ex-employee used the knowledge about EC2 instance credentials being accessible as a path to gain unauthorized access to data. In theory anyone could have exploited this vulnerability even if they had never worked for Amazon. They never say that Amazon employees had privileged credentaials that would give them unauthorized access to customer data. AWS customers that want to avoid this vulnerability should disable IMDSv1 as per https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/ https://aws.amazon.com/blogs/security/defense-in-depth-open-...
  - donavanm 6y ago
    There was zero inside knowledge and they were an ex employee at all times relevant to the incident. The EC2 instance credentials via the metadata url is public documented functionality. Its how things like the SDK “just work.” The S3 bucket policy, instance creds, and (inferred) overly permissive IAM policy is all public documented functionality. This looks like a simple case of an initial intrusion being escalated via permissive configuration and controls. There would be no story if the suspect had not been employed by AWS in the past. Disclaimer: Im a Principal jn AWS but have no direct or inside knowledge of this incident. Everything I know or have stated here is public record (eg the indictment) or public AWS docs.
- didibus 6y ago

bg24 6y ago

Can speak for AWS. Only the later. Basically the usage information for cloud resources. This constitutes the foundation for billing. BTW, this is be true for any cloud, any SAAS. There is no way an employee can look into customer data. There's enough trail inside AWS to prove that without any doubt.

KorfmannArno 6y ago
What are the measures being implemented to ensure that no employee can look into customer's data?
- yandie 6y ago
  I used to work for AWS and had to deep dive into IAM to build a feature. Basically Everytime you touch AWS your session is tagged with your credentials and has a unique ID. So everything downstream you touch has your session ID associated with it. Now say somebody from Redshift wants to access the customer's data. They will then need to access to the encryption key in KMS. The trail will be there since KMS lives in the customer's account (you can audit your own access). And for production services, human actors cannot access these keys - only production credentials can. An engineer who can log into a prod host in theory can grab the temporary credentials there but it expires in 15 minutes so your trail will be rather visible. Also access to prod host has a high bar - only senior people can do it. Now in theory somebody can coordinate with a malicious user in KMS team - but the bar is high. Also the actual master key never leaves the premise for KMS so your attack surface is very limited. Of course there are some core teams like IAM and KMS where if they become vulnerable the whole thing falls apart. But that's a big stretch for those systems since they are the core to the business.
  - jacquesm 6y ago
    This is about as bad a revelation as the original one. So the encryption key is fair game without explicit customer approval?
    - 6y ago

redredrobot 6y ago

I worked in an area where it is really hard to figure out exactly what workloads were being run and where it would have been extremely useful to know even basic things like CPU utilization patterns, network throughput patterns, etc for a specific customer. We had access to absolutely none of that information. We flew blind, relying entirely on the fact that we gave our customers enough hand-holding support that they would willingly volunteer information about their workloads so we could help them optimize it/save money. No one even attempted to get more detailed customer information AFAIK because it would have been extremely against company culture. That isn't Earning Trust or having Customer Obsession. The idea of reading data in someone's S3 bucket or inspecting what is happening inside of someone's EC2 instance in any way was unthinkable. Amazon is huge and imperfect, but from what I saw AWS takes data privacy extremely seriously.

kapilvt 6y ago

aggregated api usage stats, api client headers is often enough to identify competitor products and their traction, and is non-sensitive, coupled with account id to customers.

anonymousDan 6y ago
Do you have to use AWS to sell on Amazon?
- kapilvt 6y ago
  no