12 ms·
Cannot up vote this enough. During my time both at Retail and AWS it was perfectly normal to trawl production customer data and come up with ideas to launch com
by former-aws 6y ago
Cannot up vote this enough. During my time both at Retail and AWS it was perfectly normal to trawl production customer data and come up with ideas to launch competing products. Prices were always set lower or free offering justified as data-driven and customer obsession. I hated the gas lighting their customers and left in disgust of the company and its leadership which encourages that behavior.
- rckoepke 6y agoWhat types of AWS data would be trawled? Are we talking about data inside S3 buckets, database schemas, particular architecure styles, the fact that a product is consuming {x, y, z} amounts of cloud resources, or simply "spending $m / year" in gross?
- ShroudedNight 6y agoGiven how granular AWS billing data is, I would expect the odds to be fairly good that it alone is sufficient to make a good analysis for which third-party offerings are compelling markets. Then AWS takes their execution advantage, along with things like the lower friction that arises from first-party integration with IAM and billing, as well as not having to pay retail for the cloud resources, and it becomes very difficult to retain a moat unless you have a paradigm or perspective that is both critical to succeeding and is also incompatible with AWS culture.
- QuinnyPig 6y agoYou’re correct. It’s disturbingly detailed as far as what it reveals about architecture.
- whoisjuan 6y agoI can confidently tell you that Amazon's employees cannot see customers data inside S3 buckets or EC2 instances. They are extremely serious about that stuff since they know that will erode their customer's confidence. But there's probably other superficial business data that's helpful to evaluate that.
- KorfmannArno 6y ago1. Did you work on a team at Amazon in the likes of what user throwaway_aws mentioned? 2. What measures that you know of is Amazon implementing to make sure no employees across all teams are having access to said resources?
- whoisjuan 6y agoAs I said below this is something that they will talk a about like every freaking day. They talk about customer’s data as the most important thing to take care of. Basically is preferable to get a bullet in the head than to ever reveal or tamper with customer’s data. I cannot answer your question about who has access or not but I’m telling you what’s the culture when it comes to customer’s data. At the end of the day I was just another IC doing menial work so probably not a good reference, but that was my experience
- kaesar14 6y agoAmazon is a massive company. How can you know this with confidence? Are you in the C-Suite?
- whoisjuan 6y agoIt’s the thing they tell you the most when you work there. Like in a a obnoxious way. Most infosec training is about that. If someone has access to customer’s data for their work they have to do a bunch of extra training and do other stuff. Potentially sign some things and there’s probably a different way to authenticate. I really don’t know because I never had to do that and nobody I knew had that type of access but I heard when you do you have to put with more things.
- kaesar14 6y agoBut then what about other commenters saying that this is exactly what their sectors of the company do? Do you think it's impossible that a massive company like Amazon that controls an ungodly amount of the Internet would break those rules? Especially when the government of their home country hasn't pursued an antitrust case in God knows how long
- bg24 6y agoCan speak for AWS. Only the later. Basically the usage information for cloud resources. This constitutes the foundation for billing. BTW, this is be true for any cloud, any SAAS. There is no way an employee can look into customer data. There's enough trail inside AWS to prove that without any doubt.
- KorfmannArno 6y agoWhat are the measures being implemented to ensure that no employee can look into customer's data?
- yandie 6y agoI used to work for AWS and had to deep dive into IAM to build a feature. Basically Everytime you touch AWS your session is tagged with your credentials and has a unique ID. So everything downstream you touch has your session ID associated with it. Now say somebody from Redshift wants to access the customer's data. They will then need to access to the encryption key in KMS. The trail will be there since KMS lives in the customer's account (you can audit your own access). And for production services, human actors cannot access these keys - only production credentials can. An engineer who can log into a prod host in theory can grab the temporary credentials there but it expires in 15 minutes so your trail will be rather visible. Also access to prod host has a high bar - only senior people can do it. Now in theory somebody can coordinate with a malicious user in KMS team - but the bar is high. Also the actual master key never leaves the premise for KMS so your attack surface is very limited. Of course there are some core teams like IAM and KMS where if they become vulnerable the whole thing falls apart. But that's a big stretch for those systems since they are the core to the business.
- jacquesm 6y agoThis is about as bad a revelation as the original one. So the encryption key is fair game without explicit customer approval?
- 6y ago
- redredrobot 6y agoI worked in an area where it is really hard to figure out exactly what workloads were being run and where it would have been extremely useful to know even basic things like CPU utilization patterns, network throughput patterns, etc for a specific customer. We had access to absolutely none of that information. We flew blind, relying entirely on the fact that we gave our customers enough hand-holding support that they would willingly volunteer information about their workloads so we could help them optimize it/save money. No one even attempted to get more detailed customer information AFAIK because it would have been extremely against company culture. That isn't Earning Trust or having Customer Obsession. The idea of reading data in someone's S3 bucket or inspecting what is happening inside of someone's EC2 instance in any way was unthinkable. Amazon is huge and imperfect, but from what I saw AWS takes data privacy extremely seriously.
- kapilvt 6y agoaggregated api usage stats, api client headers is often enough to identify competitor products and their traction, and is non-sensitive, coupled with account id to customers.
- anonymousDan 6y agoDo you have to use AWS to sell on Amazon?
- kapilvt 6y agono
- deleted 6y ago[deleted]
- ajross 6y agoI think there's a difference there, though. Retail sales and reselling are parts of what most people broadly consider the "same industry". I mean, a small seller making a deal with Amazon to resell something that they know Amazon could sell on its own is at least always aware of the competition. In this case, tech investing and online retailing are not the same industry. Amazon is using a dominance in one to fund the other, which then it uses to either drive valuations of potential competitors down or to simply outcompete them. And that's a plausible antitrust problem. I'm normally not in the Amazon haters camp. Most of the time I'll defend them against the typical charges of unfair competition. Not this time. This is sketchy.
- marta_morena_25 6y ago"perfectly normal to trawl production customer data" It's not. And there are plenty of trainings inside of Amazon to make you aware of that. It is your fault, in the end, to not report your team. I have been on several teams at Amazon and this would always be an absolute no-go. It's already difficult to even get basic ideas about customer data, things that you would consider "essential" to improving the customer experience.
- icelancer 6y ago>> It is your fault, in the end, to not report your team Talk about all time gaslighting. It's the managers/directors job to ensure compliance, not normal employees.
- httpsterio 6y agoIf you see another employee committing a crime, you're obligated to report it under US law. You can be considered an accessory if you don't.
- otterley 6y agoAttorney here!* That is totally false. Conspiracy requires two elements: an agreement to commit a crime, and an act in furtherance of said crime. There is nothing unlawful about looking the other way. You might be a scumbag, but that's a different problem. The elements of criminal accessory require one to harbor, conceal, or act in such a way as to help someone avoid or escape arrest or punishment (CA law here, other states may be different). Again, merely "looking the other way" is not an act. Otherwise, anyone who merely witnessed a crime could be charged with criminal accessory. That said, corporate policy might be quite different. If I look the other way while a colleague violates customer security policies (and I'm aware of such violation), I can justifiably be fired. *Not giving legal advice, seek licensed counsel in your jurisdiciton.
- 3gg 6y agoWe need more attorneys. Attorney saves the day.
- thoraway1010 6y agoThis frankly doesn't match my experience and I have to say I find it unlikely. Before going into our AWS production S3 buckets, looking at our databases for customer lists AWS seems to be pretty careful to get an OK. Now we are being told that production customer data was normal to trawl? How in the HELL are they passing all their certs with all production data so wide open. I do customer managed keys - I mean, this is a HUGE backdoor. Either Amazon is lying about AWS security (and has fooled a bunch of others) or routinely trawling AWS customer production workloads for data is a false statement.
- starfallg 6y agoMy understanding is that Customer Managed CMK in KMS only means that the customer has control over the key operations - like rotation, key policies, IAM policies, etc. AWS still has actual control over the KMS system and full access to the HSM.
- p0rkbelly 6y agoNo. AWS has no access to your material, nor is there a code path where they could get it.
- thoraway1010 6y agoWe just had someone claiming to work for amazon who said it was "routine" to "trawl" through CUSTOMER production data. How are they trawling through all our buckets and databases without codepaths for access? Again, they aren't talking about amazon data (ie, billing, support inquiries etc). They are talking about customer production data.
- donor20 6y agoEven under this definition how in the HELL are they "routinely" trawling our production data secured by these keys. I mean, does not one think that is rediculous? This isn't amazon billing data etc (obviously I expect they analyze that carefully given they bring in billions from billing). To ROUTINELY go through AWS customer production datasets is beyond all reason.
- julianeon 6y agoI want to be careful here, as I respect that you worked at AWS (that is, most likely), while I never have, and don't know what goes inside the company. But it would be helpful if you broke that down a little more than 'trawling customer data', because at the most innocuous, if they're just looking at what's publicly selling on Amazon, what goes into sales rank, that seems acceptable, to me anyway.
- httpsterio 6y agoConsidering that OP created this account today and that they're admitting to what would be a felony and against Amazon's own privacy policy, I doubt this statement is true. Even if the customer had a misconfigured S3 bucket that was exposed to the public, it would still constitute as accessing customer data you're not meant to see. As other users have provided insight on, everything you do as an Amazon employee basically leaves a trail with your employee ID, even if you had access to private information (which you wouldn't basically because it's locked behind several layers of security). Fireable and sueable offense which Amazon would definitely not allow, let alone endorse.
- tekknik 6y agoDefinitely not defending parent here, but in this day in age many people create burner accounts specifically to avoid tying any statements back to them. It’s pretty acceptable practice to create burner accounts on HN. That said, I agree, I doubt any of these claims are true.
- swiftcoder 6y ago> everything you do as an Amazon employee basically leaves a trail with your employee ID That might be true in retail, but it wasn't anywhere close to true in AWS. When I left most engineers still had SSH access to the production hosts (and a not-insignificant portion of operations relied on that fact).
- mcqueenjordan 6y agoLeaving aside the question of what SSH access looks like today versus whenever you left... There are many easy mechanisms to audit and monitor SSH sessions. So... no?
- swiftcoder 6y agoThey weren't audited at the time (nor was there a standardised way of doing so).
- api 6y agoStallman was right: https://www.gnu.org/philosophy/who-does-that-server-really-serve.en.html https://www.gnu.org/philosophy/who-does-that-server-really-s...
- munk-a 6y agoI know it's hard to do when you're making good money and would be going against co-workers. But, if you see something, say something. This crap continues because there are too many folks that are happy to help support immoral business practices for some extra scratch. This isn't all on you in particular but when google folks started raising hell about Chinese censorship the company was forced to move. We all have the power to withdraw consent over how our labour will be used and, as software developers, we've got a strong enough employment market that we have real power to help make companies behave better - power that folks working in the warehouse are absolutely deprived of.
- burtonator 6y agoI mean the problem is corruption begets corruption. They WANT do to these things because you're going to get a massive bonus when the product you 'invented' does well because you stole the idea from an Amazon customer. Amazon needs to be properly taxed so that this crap doesn't happen anymore. The idea that they shouldn't pay taxes simply because they're large should absolutely enrage everyone.
- ampersandy 6y agoWho is saying they shouldn't be taxed because they are large? There's no 'large company' tax break. https://www.cnbc.com/2019/04/03/why-amazon-paid-no-federal-income-tax.html https://www.cnbc.com/2019/04/03/why-amazon-paid-no-federal-i... There's specific credits/exemptions in the tax code that they are able to exploit (and perhaps they can only exploit some of them _because_ they are a big company), but it really isn't about their size.
- devin 6y agoWhat you say is openly contradictory. They receive certain exemptions due to their size, but their tax bill has nothing to do with their size. ???
- ampersandy 6y ago
- rorykoehler 6y agoDidn't you anonymously tip off the customer?
- daiwaka 6y agoAs I said to throwaway -- if you are of the mind to share, i am here to listen. my email is dai.wakabayashi@nytimes.com
- neilk 6y agoCome on NYTimes! You can do better than email. Don't ask someone to admit to felonies over email. Tech employers have a LOT of power to investigate their employees' digital behavior. How about this instead: https://www.nytimes.com/tips https://www.nytimes.com/tips
- caralombardo 6y agoHi former-aws: I'm one of the reporters and would like to hear more about your experience. Mind sending me an email at cara.lombardo@wsj.com so we can connect?
- neilk 6y agocaralombardo: Please don't ask people to admit to felonies over email. That goes double for any FAANG employee; their employers have many options to surveil them. Your employer has a page listing better options https://www.wsj.com/tips https://www.wsj.com/tips
- simonebrunozzi 6y agoIn fact, I would add: do not trust a journalist that doesn't try to protect his/her source. Nothing personal, Cara Lombardo.