10 ms·

Lessons from 10 Years of Amazon Web Services

169 points by yarapavan 11y ago

yarapavan 11y ago
James Hamilton listed some of the highlights over the last decade at his blog - http://perspectives.mvdirona.com/2016/03/a-decade-of-innovation/ http://perspectives.mvdirona.com/2016/03/a-decade-of-innovat...
- jcrites 11y ago
  James has put out a lot of great material. He's given a talk series and wrote a conference paper called "On Designing and Deploying Internet-Scale Services" which is a trove of valuable design insights: https://www.usenix.org/legacy/event/lisa07/tech/full_papers/hamilton/hamilton_html/ https://www.usenix.org/legacy/event/lisa07/tech/full_papers/... (somewhat condensed) Slides from a talk: http://mvdirona.com/jrh/talksAndPapers/JamesRH_AmazonDev.pdf http://mvdirona.com/jrh/talksAndPapers/JamesRH_AmazonDev.pdf He writes some of the more practical and useful advice I've seen about how to run a successful business based on high-scale distributed systems. One of his mnemonics that's stuck with me is the "Four Rs" of recovery-oriented computing: Restart, Reboot, Reimage, Replace (slide 6 in PDF). A human shouldn't be engaged to troubleshoot a problem until the platform has first tried in successing to restart the software, reboot the OS, re-image the machine, and finally replace the hardware entirely. After these auto-recovery steps have failed, only then is it time to engage a human. He describes the connection between these techniques and significantly lower operations costs.
axelfontaine 11y ago
"A good litmus test has been that if you need to SSH into a server or an instance, you still have more to automate." Amen. We couldn't agree more: https://boxfuse.com/blog/no-ssh https://boxfuse.com/blog/no-ssh
- bluecmd 11y ago
  So, with No SSH - how do you debug that one-off problem that is only on machine abc12? I'm not talking about mutating, I'm talking about attaching gdb to the process while it's handling requests. I'm talking about collecting CPU profiling information from production. Stuff like that.
  - mhw 11y ago
    Immutable Infrastructure is the modern day "have you tried turning it off and back on again?"
  - autotune 11y ago
    You could probably have a central logging service like graylog or logstash that automatically collects that information and just sort through the logs that way.
    - AznHisoka 11y ago
      What if that 1 server ran out of file handles and can no longer log to anywhere?
      - deleted 11y ago
        [deleted]
      - xxpor 11y ago
        Take the server out back and shoot it.
      - autotune 11y ago
        Launch a copy of that server with a new image (which I would hope is under a load balancer), update the sysctl.conf file template with a larger fs.file-max limit in whatever config management system you use for that instance, resume logging under freshly cloned instance.
        dsp1234 11y ago
- LinuxBender 11y ago
  Proposal: Instead of ssh vs. no-ssh debate, how about: * Use SSH AllowGroups to limit logins to the development manager/lead or on-call engineer * with a policy that for each time you have to call them beyond N times per week, you have to buy their team a round of drinks of their choosing? Deal?
jimbokun 11y ago
"Primitives not frameworks" This is why Amazon beat Google in the "as a Service" space. Google provided a framework with AppEngine, but customers wanted the flexibility of the primitives offered by AWS.
- nopzor 11y ago
  I think that Amazon being first in 2005/2006 helped a lot, too. Google now provides "primitives" through GCE. I think Google realizes that the future is GCE not GAE.
  - vgt 11y ago
    I think there's tremendous value still in GAE. To this day the fact that a company the size of Snapchat is able to operate on a PaaS, with virtually no ops people, is STILL a game-changer. The fact that it's still true 3+ years in is equally as impressive. The narrative shifted away from PaaS for a number of reasons, but the value prop is still present. (Disclaimer: work on BigQuery, a DBaaS, which is, in my opinion, is clearly the future)
  - fensterblick 11y ago
    But Amazon wasn't the first. The Sun Grid was first (launched in March 2006).
    - nopzor 11y ago
      Sun Grid was more about batch processing across many machines, rather than individual VMs, wasn't it? Plenty of people in the web hosting industry offered competitive x86 linux instances or dedicated servers, way prior to 2006. I think AMZN deserves credit for delivering a price compelling option that delivered (1) elasticity (per hour billing) and (2) automation (API, available in minutes), among other things.
- eranation 11y ago
  Exactly Offer IaaS first, the PaaS. Google had it backward. While AWS does have a small vendor lock in too, in the end it's still recognizable , I mean it's doable for example to move an AWS architecture to open stack without a major code rewrite. If you built your web app on GAE and GWT however, your investment to move it out of Google will be much bigger.
rodionos 11y ago
You have to give full credit to AWS for inventing previously unknown pricing models. In some cases, it was actually the pricing model that created new use cases. Glacier and S3 comes to mind.
- jeffbarr 11y ago
  That's a very important observation. When I describe AWS to audiences I always make clear that it is a combination of a technology and a business model.

169 points by yarapavan 11y ago

James Hamilton listed some of the highlights over the last decade at his blog - http://perspectives.mvdirona.com/2016/03/a-decade-of-innovation/ http://perspectives.mvdirona.com/2016/03/a-decade-of-innovat...

jcrites 11y ago
James has put out a lot of great material. He's given a talk series and wrote a conference paper called "On Designing and Deploying Internet-Scale Services" which is a trove of valuable design insights: https://www.usenix.org/legacy/event/lisa07/tech/full_papers/hamilton/hamilton_html/ https://www.usenix.org/legacy/event/lisa07/tech/full_papers/... (somewhat condensed) Slides from a talk: http://mvdirona.com/jrh/talksAndPapers/JamesRH_AmazonDev.pdf http://mvdirona.com/jrh/talksAndPapers/JamesRH_AmazonDev.pdf He writes some of the more practical and useful advice I've seen about how to run a successful business based on high-scale distributed systems. One of his mnemonics that's stuck with me is the "Four Rs" of recovery-oriented computing: Restart, Reboot, Reimage, Replace (slide 6 in PDF). A human shouldn't be engaged to troubleshoot a problem until the platform has first tried in successing to restart the software, reboot the OS, re-image the machine, and finally replace the hardware entirely. After these auto-recovery steps have failed, only then is it time to engage a human. He describes the connection between these techniques and significantly lower operations costs.

axelfontaine 11y ago

"A good litmus test has been that if you need to SSH into a server or an instance, you still have more to automate." Amen. We couldn't agree more: https://boxfuse.com/blog/no-ssh https://boxfuse.com/blog/no-ssh

bluecmd 11y ago
So, with No SSH - how do you debug that one-off problem that is only on machine abc12? I'm not talking about mutating, I'm talking about attaching gdb to the process while it's handling requests. I'm talking about collecting CPU profiling information from production. Stuff like that.
- mhw 11y ago
  Immutable Infrastructure is the modern day "have you tried turning it off and back on again?"
- autotune 11y ago
  You could probably have a central logging service like graylog or logstash that automatically collects that information and just sort through the logs that way.
  - AznHisoka 11y ago
    What if that 1 server ran out of file handles and can no longer log to anywhere?
    - deleted 11y ago
      [deleted]
    - xxpor 11y ago
      Take the server out back and shoot it.
    - autotune 11y ago
      Launch a copy of that server with a new image (which I would hope is under a load balancer), update the sysctl.conf file template with a larger fs.file-max limit in whatever config management system you use for that instance, resume logging under freshly cloned instance.
      - dsp1234 11y ago
LinuxBender 11y ago
Proposal: Instead of ssh vs. no-ssh debate, how about: * Use SSH AllowGroups to limit logins to the development manager/lead or on-call engineer * with a policy that for each time you have to call them beyond N times per week, you have to buy their team a round of drinks of their choosing? Deal?

jimbokun 11y ago

"Primitives not frameworks" This is why Amazon beat Google in the "as a Service" space. Google provided a framework with AppEngine, but customers wanted the flexibility of the primitives offered by AWS.

nopzor 11y ago
I think that Amazon being first in 2005/2006 helped a lot, too. Google now provides "primitives" through GCE. I think Google realizes that the future is GCE not GAE.
- vgt 11y ago
  I think there's tremendous value still in GAE. To this day the fact that a company the size of Snapchat is able to operate on a PaaS, with virtually no ops people, is STILL a game-changer. The fact that it's still true 3+ years in is equally as impressive. The narrative shifted away from PaaS for a number of reasons, but the value prop is still present. (Disclaimer: work on BigQuery, a DBaaS, which is, in my opinion, is clearly the future)
- fensterblick 11y ago
  But Amazon wasn't the first. The Sun Grid was first (launched in March 2006).
  - nopzor 11y ago
    Sun Grid was more about batch processing across many machines, rather than individual VMs, wasn't it? Plenty of people in the web hosting industry offered competitive x86 linux instances or dedicated servers, way prior to 2006. I think AMZN deserves credit for delivering a price compelling option that delivered (1) elasticity (per hour billing) and (2) automation (API, available in minutes), among other things.
eranation 11y ago
Exactly Offer IaaS first, the PaaS. Google had it backward. While AWS does have a small vendor lock in too, in the end it's still recognizable , I mean it's doable for example to move an AWS architecture to open stack without a major code rewrite. If you built your web app on GAE and GWT however, your investment to move it out of Google will be much bigger.

rodionos 11y ago

You have to give full credit to AWS for inventing previously unknown pricing models. In some cases, it was actually the pricing model that created new use cases. Glacier and S3 comes to mind.

jeffbarr 11y ago
That's a very important observation. When I describe AWS to audiences I always make clear that it is a combination of a technology and a business model.