9 ms·

Disclaimer: I worked at AWS from 2012-2020. I am not a current employee. > I came to think of Amazon and Google as the providers of the very infrastructure of

by ragona 6y ago

Disclaimer: I worked at AWS from 2012-2020. I am not a current employee. > I came to think of Amazon and Google as the providers of the very infrastructure of the internet, so embedded in the architecture of the digital world that even their competitors had to rely on their services. This is the right take. I disagree with the disappointment around this article. It’s about the fact that tech giants are running the infrastructure of the internet. Its the federal highway program of the 21st century. It’s a critical piece of public infrastructure. I can’t even begin to wrap my head around the death and mayhem that would ensue if AWS and Google simply packed up and turned off the data centers. When we consider the future and safety of our country we should keep these things in mind. Imagine an unusually well-attended AWS all-hands was hit by a freak meteor. What do you even do? Take enough engineers off the oncall rotation and things will eventually fall over. I think the government would actually need to take emergency action at a level we don’t understand to prevent the collapse of our financial system beyond god knows what else. It would be utter mayhem. Globally. That said, no one takes this more seriously than AWS. I really believe this. I sat in meetings regarding how to recover if we lost entire states, like literally what if we lose the Virginia to a catastrophe. (I worked on the Key Management Service or KMS which underlies much of the rest of the infrastructure.) But it is good and correct that our representatives are questioning our tech giants about their role in the future of our civilization. They’re playing perhaps the most critical role, and that should scare you. I actually believe in the benevolence of AWS; I spent nearly a decade there. Do you?

mbostleman 6y ago
Sure, I'll believe it. I've not heard enough to the contrary to not believe it, and you make a good recommendation. But benevolence is transient and brittle. I don't want to depend on it.
- ragona 6y ago
  > But benevolence is transient and brittle. I don't want to depend on it. Yup, exactly.
ls612 6y ago
That makes me wonder, how did AWS or other big cloud providers handle the stay at home orders in March? Did they just keep having people go in?
- gberger 6y ago
  I'm pretty sure data center workers were considered essential workers.
- ragona 6y ago
  The data center operations team is probably still in the building. But the software engineering teams which is what I’m familiar with are all working from home. We don’t have to go to the data centers anyway usually.
- ckozlowski 6y ago
  Except where absolutely necessary, we've been WFH since March. We were allowed to expense some things for setting up a home office. The few physical workspaces that are still active out of necessity have a ton of sanitation measures in place. Regular cleanings, temp sensors, etc. I knew someone was paying attention to detail when those "foot handles" started appearing on the bottoms of the bathroom entry doors. A lot of my colleagues and myself have been busier it feels than any time prior. There is honestly a real sense that a lot of people are suddenly depending on us more than ever. It does give the sense that we're critical infrastructure, and the leadership communicates and appreciates as such.
  - bosie 6y ago
    > the leadership communicates and appreciates as such. What does this mean, especially the latter?
- foenix 6y ago
  not OP, but my AWS buddy was told to work from home back in February if I recall correctly.
ckozlowski 6y ago
Absolutely. So long as the employees who run it believe in upholding that benevolence themselves. Disclaimer: I'm a current TAM with AWS.
- DoctorNick 6y ago
  As Snowden says, tech workers are complicit in how their companies hurt society: https://www.vice.com/en_us/article/wxqx8q/snowden-tech-workers-are-complicit-in-how-their-companies-hurt-society https://www.vice.com/en_us/article/wxqx8q/snowden-tech-worke...
  - spaced-out 6y ago
    If that's true, then so are those that invest in big tech companies.
    - DoctorNick 6y ago
      Absolutely. The material interests of the shareholders ultimately drive the decisions that these companies make. If you invest in them, you are complicit.
  - Nasrudith 6y ago
    The real problem is "hurting society" is a bullshit charge which lets you call yourself a hero for making Socrates poison himself. The sophistry is all brain-hacking bullshit best dealt with like a poisoned drink - thrown in the face of the server.
endymi0n 6y ago
Companies are way faster, focussed and efficient than any form of non-profit or governmental organization in building infrastructure that supports their cause. I'm eternally grateful to Google for HTTP/2, Chrome, Web Vitals, Angular and so many other puzzle pieces without which the whole Internet wouldn't be what it is today. It showed foresight, wisdom, excellence, responsibility and stewardship. The problem starts where growth ends, and all growth needs to end somewhere. There are only so many people on the planet with 24h a day to spend. When that problem starts, a corporation still has to maximize shareholder value, and that's where the formally great stewardship gets endangered. And then Google used its web browser, standards and search monopoly in order to - shove the open-but-really-not-open AMP "standard" down everybody's throat - eat up any kinds of comparison competitors by stealing their content into the featured snippets - ... So going back to your question, I'd still mostly trust in the benevolence of AWS — but I have zero doubts this benevolence will go away as soon as there is an opportunity to crush a competitor by ignoring that benevolence. To quote Jeff Bezos here: "Your margin is my opportunity".
- pwdisswordfish4 6y ago
  > in building infrastructure that supports their cause. Emphasis on this part.
  - dwaltrip 6y ago
    To fully spell it out, the causes of a company don't always align with those of everyone else. Profit motive alone is not sufficient to adequately align companies with the interests of broader society. Which is why a market that is completely "free", where actors are not beholden to any laws or regulations, is not a good idea.
- russdpale 6y ago
  > Companies are way faster, focussed and efficient than any form of non-profit or governmental organization in building infrastructure that supports their cause. So what? What does that have to do with anything? I would disagree with this statement wholeheartedly. Private enterprise did not save the midwest from the dust storms, government did. Private enterprise did not save the world from the Nazi's, government did. Private enterprise will not save us from global warming, government will. Indeed, all of these problems are generally started by private monied interest. Without the government, those companies can't even operate, so your entire premise is totally backwards.
treis 6y ago
>I can’t even begin to wrap my head around the death and mayhem that would ensue if AWS and Google simply packed up and turned off the data centers. There's lots of companies out there that would cause mayhem if they simply closed up shop. But there's plenty of alternatives out there and self hosting is still an option. We can do without AWS. It would take more than a day to migrate but we could do it.
- loriverkutya 6y ago
  Sorry, you seriously have no idea what you are talking about. Self hosting our company infrastructure would require hundreds of physical servers. That's the easy part. Put those servers into a room, where they can work, they have power, network etc. Yes, we could move to self hosting. Yes, the company is dead before the servers are shipped, if amazon goes down everywhere and not recover. If they lose a region we are fine. And we are a medium company.
  - treis 6y ago
    ? I'm not sure why the hostility or what exactly you disagree with. I said that we could migrate off of AWS but it'd take more than a day. You say that your company can migrate off AWS but it would take more than a day. Where's the disagreement?
    - ragona 6y ago
      I'm curious how many companies would survive it at all. Sure, you could _migrate_ off of AWS if AWS was still alive. Could you restore your data and start your business from scratch if you lost everything? I suspect many rather large businesses could not.
  - luckylion 6y ago
    So it's "we have, without necessity, put all our eggs in one basket, so now this basket should be treated as a utility so we don't have to spent a bit more money & time to protect us against unforeseen consequences of that basket breaking"? What if there's an error and Amazon bans your account, do you just shut down and give up? There's no contingency plan whatsoever?
InvisibleCities 6y ago
>I actually believe in the benevolence of AWS; I spent nearly a decade there. Do you? I have no reason to doubt the benevolence of AWS at present, but how much faith do you have in the perpetuity of that benevolence? There is a tremendous power imbalance between Amazon/Google/Microsoft and the companies that depend on them for infrastructure, and the nature of capitalism and the declining rate of profitability means that eventually, the markets will demand for that power imbalance to be exploited economically. When the Manorial system emerged in Europe, it was a largely benevolent force offering stability, security, and community in the wake of the collapsing Western Roman Empire - within a few generations it morphed into 1000 years of serfdom.
- ragona 6y ago
  > how much faith do you have in the perpetuity of that benevolence None! Literally none. This is a huge risk for our country, and we should thinking about what it means for the next 1000 years, not the next 10. Bezos appears to truly, truly believe in Customer Obsession. The culture definitely softened while I was there though, and I can't imagine how it'll change in 200 years. I can't guarantee his great grandkids won't be dicks. I don't really want to live in the corporatist city state of Seattle By Amazon with its "benevolent" AI overlord and its drone stazi in 2234, ya know?
ssivark 6y ago
Did you just make a strong argument for why these services ought to be “nationalized” like public infrastructure?
- rwnspace 6y ago
  It's a natural good like water at this point. And the psycho-social mechanistic designs which go into what you consume are much more dangerous than added fluoride.
- QuercusMax 6y ago
  That's a great idea! At this point, you could totally say that Larry, Sergey, Jeff, and Mark have all won at capitalism. Let's cash them out and give their shares to the people.
  - ragona 6y ago
    I love this. Imagine, your company hits $1T. POTUS gets you on stage. They tell you you did SO GOOD. We're really proud of you. We're gonna make you a statue in the Hall of Really Good Capitalists, and we have the best news. You are SO GOOD at this that we're gonna let you do it again. You're poor now, but we're gonna put your name in history books! If you get this rich again on hardcore mode we will make your statue bigger!
    - QuercusMax 6y ago
      I'm not suggesting taking away their money. They built a trillion dollar company that's vital to the national economy. Yay! Now we (the people) buy it from them and it belongs to all of us.
    - zanderwohl 6y ago
      It's like a game of cookie clicker. Once you get all the cookies, you ascend and start clicking again.
- ragona 6y ago
  Well I'm glad you thought my argument was strong! Yes, that is what I believe. I see a path where after Amazon creates something that becomes a de facto public good they hand it over to a highly competent government agency full of the world's best operational engineers. Those people turn it into a public works project, and it has legendary 25-nines uptime. Amazon engineers are excited to see their work become part of the fabric of the Galactic Alliance of Good and Kind Beings and no one does oncall like a Quarkian, am I right, 700-hour sleep/wake cycles how cool is that, love working with them. Do I see a path to get there from here? Uh, not at the moment. We're still working on "hitting each other is bad" so I'm feeling pretty depressed about our outlook.
screye 6y ago
> I actually believe in the benevolence of AWS; I spent nearly a decade there. Do you? As many problems as capitalism has, often monetary goals align perfectly with the 'ethical' action. In my experience, capitalistic companies that optimize for long term survival (as is characteristic of Amazon) usually act in a less malevolent manner. However, any company that sees a revolving door of CEOs, will be seen to optimize for short term gains at the cost of consumer friendliness.
deleted 6y ago
[deleted]
mcintyre1994 6y ago
I didn't realise that KMS underlay AWS' internal infrastructure, literally the same KMS service that's exposed to users? Do you know if AWS' internal infrastructure is actually documented publicly in terms of how services fit together internally anywhere? Or is that all considered secret beyond what's exposed to users?
- ragona 6y ago
  Yeah, this stuff isn't secret. Note how if you encrypt your S3 bucket it uses a KMS CMK; that's an dependency between S3 and KMS, and yep it's the same KMS. This is true for probably over a hundred AWS services that rely on KMS; it's a "foundational" service. (As is S3.)
dumbfoundded 6y ago
No, I don't believe any piece of critical infrastructure works if privately owned. I'm glad the highways aren't privately owned. I'm glad USPS exists so it can deliver mail unprofitably to my rural community. I'm glad water isn't privatized. The industries that exist privately in such crucial sectors must also undergo heavy regulation. I have no misgivings that Amazon and Google at their highest levels, pretty much only work out of private interest. It's nice when they align with the public interest but as soon as they don't, I fear what will happen. Energy companies have no problems with polluting the environment. Between the environmental, political, and military consequences, it seems destructive when a profit motive has leverage over the rest of the country.
- pkaye 6y ago
  > No, I don't believe any piece of critical infrastructure works if privately owned. I'm glad the highways aren't privately owned. I'm glad USPS exists so it can deliver mail unprofitably to my rural community. I'm glad water isn't privatized. Aren't some highways in the east coast privately run toll roads? And right now Trump and the Republicans are trying to gut the USPS.
  - dumbfoundded 6y ago
    I don't support highway tolls b/c they are regressive taxes. Trump and the Republicans have been trying to gut USPS for years. Now they seem extremely urgent to advance their plans.
simonebrunozzi 6y ago
> I actually believe in the benevolence of AWS; I spent nearly a decade there. Do you? I do not (I spent 6 years at AWS, 2008-2014), but don't believe in "evil" either; to support your point, though, the "alternative" to AWS is certainly worse. I agree with your point that taking AWS away would cause a lot of problems (and actually, Azure and GCP are the redundancy to AWS; hopefully we don't get any meteor to hit that crowd, but if we do, hopefully there's only one, not 3).

by ragona 6y ago

mbostleman 6y ago

Sure, I'll believe it. I've not heard enough to the contrary to not believe it, and you make a good recommendation. But benevolence is transient and brittle. I don't want to depend on it.

ragona 6y ago
> But benevolence is transient and brittle. I don't want to depend on it. Yup, exactly.

ls612 6y ago

That makes me wonder, how did AWS or other big cloud providers handle the stay at home orders in March? Did they just keep having people go in?

gberger 6y ago
I'm pretty sure data center workers were considered essential workers.
ragona 6y ago
The data center operations team is probably still in the building. But the software engineering teams which is what I’m familiar with are all working from home. We don’t have to go to the data centers anyway usually.
ckozlowski 6y ago
Except where absolutely necessary, we've been WFH since March. We were allowed to expense some things for setting up a home office. The few physical workspaces that are still active out of necessity have a ton of sanitation measures in place. Regular cleanings, temp sensors, etc. I knew someone was paying attention to detail when those "foot handles" started appearing on the bottoms of the bathroom entry doors. A lot of my colleagues and myself have been busier it feels than any time prior. There is honestly a real sense that a lot of people are suddenly depending on us more than ever. It does give the sense that we're critical infrastructure, and the leadership communicates and appreciates as such.
- bosie 6y ago
  > the leadership communicates and appreciates as such. What does this mean, especially the latter?
foenix 6y ago
not OP, but my AWS buddy was told to work from home back in February if I recall correctly.

ckozlowski 6y ago

Absolutely. So long as the employees who run it believe in upholding that benevolence themselves. Disclaimer: I'm a current TAM with AWS.

DoctorNick 6y ago
As Snowden says, tech workers are complicit in how their companies hurt society: https://www.vice.com/en_us/article/wxqx8q/snowden-tech-workers-are-complicit-in-how-their-companies-hurt-society https://www.vice.com/en_us/article/wxqx8q/snowden-tech-worke...
- spaced-out 6y ago
  If that's true, then so are those that invest in big tech companies.
  - DoctorNick 6y ago
    Absolutely. The material interests of the shareholders ultimately drive the decisions that these companies make. If you invest in them, you are complicit.
- Nasrudith 6y ago
  The real problem is "hurting society" is a bullshit charge which lets you call yourself a hero for making Socrates poison himself. The sophistry is all brain-hacking bullshit best dealt with like a poisoned drink - thrown in the face of the server.

endymi0n 6y ago

Companies are way faster, focussed and efficient than any form of non-profit or governmental organization in building infrastructure that supports their cause. I'm eternally grateful to Google for HTTP/2, Chrome, Web Vitals, Angular and so many other puzzle pieces without which the whole Internet wouldn't be what it is today. It showed foresight, wisdom, excellence, responsibility and stewardship. The problem starts where growth ends, and all growth needs to end somewhere. There are only so many people on the planet with 24h a day to spend. When that problem starts, a corporation still has to maximize shareholder value, and that's where the formally great stewardship gets endangered. And then Google used its web browser, standards and search monopoly in order to - shove the open-but-really-not-open AMP "standard" down everybody's throat - eat up any kinds of comparison competitors by stealing their content into the featured snippets - ... So going back to your question, I'd still mostly trust in the benevolence of AWS — but I have zero doubts this benevolence will go away as soon as there is an opportunity to crush a competitor by ignoring that benevolence. To quote Jeff Bezos here: "Your margin is my opportunity".

pwdisswordfish4 6y ago
> in building infrastructure that supports their cause. Emphasis on this part.
- dwaltrip 6y ago
  To fully spell it out, the causes of a company don't always align with those of everyone else. Profit motive alone is not sufficient to adequately align companies with the interests of broader society. Which is why a market that is completely "free", where actors are not beholden to any laws or regulations, is not a good idea.
russdpale 6y ago
> Companies are way faster, focussed and efficient than any form of non-profit or governmental organization in building infrastructure that supports their cause. So what? What does that have to do with anything? I would disagree with this statement wholeheartedly. Private enterprise did not save the midwest from the dust storms, government did. Private enterprise did not save the world from the Nazi's, government did. Private enterprise will not save us from global warming, government will. Indeed, all of these problems are generally started by private monied interest. Without the government, those companies can't even operate, so your entire premise is totally backwards.

treis 6y ago

>I can’t even begin to wrap my head around the death and mayhem that would ensue if AWS and Google simply packed up and turned off the data centers. There's lots of companies out there that would cause mayhem if they simply closed up shop. But there's plenty of alternatives out there and self hosting is still an option. We can do without AWS. It would take more than a day to migrate but we could do it.

loriverkutya 6y ago
Sorry, you seriously have no idea what you are talking about. Self hosting our company infrastructure would require hundreds of physical servers. That's the easy part. Put those servers into a room, where they can work, they have power, network etc. Yes, we could move to self hosting. Yes, the company is dead before the servers are shipped, if amazon goes down everywhere and not recover. If they lose a region we are fine. And we are a medium company.
- treis 6y ago
  ? I'm not sure why the hostility or what exactly you disagree with. I said that we could migrate off of AWS but it'd take more than a day. You say that your company can migrate off AWS but it would take more than a day. Where's the disagreement?
  - ragona 6y ago
    I'm curious how many companies would survive it at all. Sure, you could _migrate_ off of AWS if AWS was still alive. Could you restore your data and start your business from scratch if you lost everything? I suspect many rather large businesses could not.
- luckylion 6y ago
  So it's "we have, without necessity, put all our eggs in one basket, so now this basket should be treated as a utility so we don't have to spent a bit more money & time to protect us against unforeseen consequences of that basket breaking"? What if there's an error and Amazon bans your account, do you just shut down and give up? There's no contingency plan whatsoever?

InvisibleCities 6y ago

>I actually believe in the benevolence of AWS; I spent nearly a decade there. Do you? I have no reason to doubt the benevolence of AWS at present, but how much faith do you have in the perpetuity of that benevolence? There is a tremendous power imbalance between Amazon/Google/Microsoft and the companies that depend on them for infrastructure, and the nature of capitalism and the declining rate of profitability means that eventually, the markets will demand for that power imbalance to be exploited economically. When the Manorial system emerged in Europe, it was a largely benevolent force offering stability, security, and community in the wake of the collapsing Western Roman Empire - within a few generations it morphed into 1000 years of serfdom.

ragona 6y ago
> how much faith do you have in the perpetuity of that benevolence None! Literally none. This is a huge risk for our country, and we should thinking about what it means for the next 1000 years, not the next 10. Bezos appears to truly, truly believe in Customer Obsession. The culture definitely softened while I was there though, and I can't imagine how it'll change in 200 years. I can't guarantee his great grandkids won't be dicks. I don't really want to live in the corporatist city state of Seattle By Amazon with its "benevolent" AI overlord and its drone stazi in 2234, ya know?

ssivark 6y ago

Did you just make a strong argument for why these services ought to be “nationalized” like public infrastructure?

rwnspace 6y ago
It's a natural good like water at this point. And the psycho-social mechanistic designs which go into what you consume are much more dangerous than added fluoride.
QuercusMax 6y ago
That's a great idea! At this point, you could totally say that Larry, Sergey, Jeff, and Mark have all won at capitalism. Let's cash them out and give their shares to the people.
- ragona 6y ago
  I love this. Imagine, your company hits $1T. POTUS gets you on stage. They tell you you did SO GOOD. We're really proud of you. We're gonna make you a statue in the Hall of Really Good Capitalists, and we have the best news. You are SO GOOD at this that we're gonna let you do it again. You're poor now, but we're gonna put your name in history books! If you get this rich again on hardcore mode we will make your statue bigger!
  - QuercusMax 6y ago
    I'm not suggesting taking away their money. They built a trillion dollar company that's vital to the national economy. Yay! Now we (the people) buy it from them and it belongs to all of us.
  - zanderwohl 6y ago
    It's like a game of cookie clicker. Once you get all the cookies, you ascend and start clicking again.
ragona 6y ago
Well I'm glad you thought my argument was strong! Yes, that is what I believe. I see a path where after Amazon creates something that becomes a de facto public good they hand it over to a highly competent government agency full of the world's best operational engineers. Those people turn it into a public works project, and it has legendary 25-nines uptime. Amazon engineers are excited to see their work become part of the fabric of the Galactic Alliance of Good and Kind Beings and no one does oncall like a Quarkian, am I right, 700-hour sleep/wake cycles how cool is that, love working with them. Do I see a path to get there from here? Uh, not at the moment. We're still working on "hitting each other is bad" so I'm feeling pretty depressed about our outlook.

screye 6y ago

> I actually believe in the benevolence of AWS; I spent nearly a decade there. Do you? As many problems as capitalism has, often monetary goals align perfectly with the 'ethical' action. In my experience, capitalistic companies that optimize for long term survival (as is characteristic of Amazon) usually act in a less malevolent manner. However, any company that sees a revolving door of CEOs, will be seen to optimize for short term gains at the cost of consumer friendliness.

deleted 6y ago

[deleted]

mcintyre1994 6y ago

I didn't realise that KMS underlay AWS' internal infrastructure, literally the same KMS service that's exposed to users? Do you know if AWS' internal infrastructure is actually documented publicly in terms of how services fit together internally anywhere? Or is that all considered secret beyond what's exposed to users?

ragona 6y ago
Yeah, this stuff isn't secret. Note how if you encrypt your S3 bucket it uses a KMS CMK; that's an dependency between S3 and KMS, and yep it's the same KMS. This is true for probably over a hundred AWS services that rely on KMS; it's a "foundational" service. (As is S3.)

dumbfoundded 6y ago

No, I don't believe any piece of critical infrastructure works if privately owned. I'm glad the highways aren't privately owned. I'm glad USPS exists so it can deliver mail unprofitably to my rural community. I'm glad water isn't privatized. The industries that exist privately in such crucial sectors must also undergo heavy regulation. I have no misgivings that Amazon and Google at their highest levels, pretty much only work out of private interest. It's nice when they align with the public interest but as soon as they don't, I fear what will happen. Energy companies have no problems with polluting the environment. Between the environmental, political, and military consequences, it seems destructive when a profit motive has leverage over the rest of the country.

pkaye 6y ago
> No, I don't believe any piece of critical infrastructure works if privately owned. I'm glad the highways aren't privately owned. I'm glad USPS exists so it can deliver mail unprofitably to my rural community. I'm glad water isn't privatized. Aren't some highways in the east coast privately run toll roads? And right now Trump and the Republicans are trying to gut the USPS.
- dumbfoundded 6y ago
  I don't support highway tolls b/c they are regressive taxes. Trump and the Republicans have been trying to gut USPS for years. Now they seem extremely urgent to advance their plans.

simonebrunozzi 6y ago

> I actually believe in the benevolence of AWS; I spent nearly a decade there. Do you? I do not (I spent 6 years at AWS, 2008-2014), but don't believe in "evil" either; to support your point, though, the "alternative" to AWS is certainly worse. I agree with your point that taking AWS away would cause a lot of problems (and actually, Azure and GCP are the redundancy to AWS; hopefully we don't get any meteor to hit that crowd, but if we do, hopefully there's only one, not 3).