8 ms·

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

by jmccree 12y ago

Nick-Craver 12y ago
The cloud is awesome if you have zero interest whatsoever in hardware. It's not without trade-offs (nor is the other direction), and too many for us - but if it works for you then great I say. We obviously feel very different, and are just doing what works best for us.
- balls187 12y ago
  > We obviously feel very different, and are just doing what works best for us. And doing it very well, if I may say so.
- beachstartup 12y ago
  most of the public cloud cheer-leading is just people rationalizing to themselves what an awesome decision they made deploying on aws onto a billion tiny instances or whatever. and for lots of folks, it probably is pretty awesome. however, i've seen very, very few people compare actual before/after $ figures on hn. when it comes time to show your cards everyone gets cold feet, either because they 1. don't have any idea because they aren't the ones paying for it or 2. don't have a baseline for comparison and are just paying whatever amazon asks or 3. it ended up costing 2-4x as much on amazon when your bill is $5k/month 2-4x isn't that big of a deal. when it's $100k+ a month, it becomes a really big deal.
rdoherty 12y ago
I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a launch configuration and updating an auto-scaling group. Maybe an hour of my time to update configs, verify and deploy. Updating something like a database or caching servers is more work for sure, but with 0 time needed to get to the DC, unpack, rack and configure servers you do save time with 'the cloud'. I get that you do pay more for EC2 instances, especially if you keep hardware for 4 years. But AWS prices drop every year or two along with (generally) faster versions of software so your overall costs do drop. How many ops employees would you need for a fleet of 500 servers in a datacenter? We managed it all with 4 people with AWS.
- bri3d 12y ago
  The Stack Exchange philosophy is that because they can buy truly mega hardware (each one of those two blade chassis they bought has 72 cores and 1.4TB of RAM, remember!), they don't need those 500 servers to start with. Plus the hardware is an asset and you get to depreciate it. Everywhere I've ever worked we've had the "big spreadsheet" of projected cloud costs, projected ops costs, and hardware costs. In general the "scale horizontally" philosophy will favor the cloud while the "scale vertical" philosophy still seems to favor owned hardware in local datacenters. Which is superior is a crazy, long-standing debate with no clear answer.
  - MrBuddyCasino 12y ago
    Can you elaborate? I thought the answer to that question was to scale up if you can, because its much simpler and therefore cheaper. Similar to how you don't give up ACID unless the scale you're working at doesn't permit it anymore.
    - thirsteh 12y ago
      > I thought the answer to that question was to scale up if you can, because its much simpler and therefore cheaper ✱cough✱ SQL Server licensing fees ✱cough✱
      - bri3d 12y ago
        That's a good point that I overlooked in my post - I'm sure this is a huge consideration for Stack Exchange. For what it's worth Basecamp also evangelize the "scale up" approach and they're on an open-source stack.
    - bri3d 12y ago
      There's never really a "one size fits all" answer, which is why it's a long-running debate and depends heavily on the product. Scaling horizontally can let you use smaller, cheaper hardware on average and burst to higher capacity more easily if you need to, at the expense of a lot of complexity. It also (done right, which is rare) tends to gain you a greater degree of fault tolerance, since hardware instances become rapidly-replaceable commodities. Most web apps have spiky but relatively predictable load. For example, a typical enterprise SaaS startup gets more traffic during work hours than on weekends. For these companies the complexity of developing a horizontally scaled architecture can be offset by the decreased cost of buying really big machines for peak load and then scaling back to a couple small instances for periods of below-average load. That's (ostensibly) why AWS exists in the first place: Amazon had to buy a lot of peak capacity for Black Friday and Christmas and found it going unused the rest of the year. They never meant to sell their excess capacity, but they realized the tools that they built to dynamically scale their infrastructure were valuable to others. Plus, a lot of work is offline data analytics, ETL, and so on. It's very cost effective to scale these workloads horizontally on-demand - spin up extra workers to run your reporting each hour/night and keep costs down the rest of the time when you don't need the capacity. On the flip side, companies like Stack Exchange and Basecamp have high, relatively stable traffic worldwide. For companies like this it makes more sense to scale vertically - if they were in the cloud, they would never scale down or shut down their instances anyway. Personally, I agree that horizontal scalability is oversold and most people can, indeed, scale up instead of out. However, a lot of plenty smart people disagree with me and have valid reasons to scale horizontally, too.
jimwalsh 12y ago
Actually, occasionally you do need to think about it. Reference the various times AWS emails customers about unplanned outages that need to occur because of hardware issues/patching/etc. Cloud is great, but there str still plenty of reasons to run your own datacenter. Yes for many startups it might not make sense, but at a certain sized company/application it can easily make sense.
- ak217 12y ago
  AWS just announced a few days ago that their latest Xen patch will be deployed through a live update to their hypervisor kernel, and that going forward they expect patches like this to be rolled out live. The real upside of AWS is that they have relentlessly pursued and killed off reasons for you to care about things like this. They've eliminated points of failure in their infrastructure and given operators a wealth of tools to ensure their apps stay up through any update or event (AZ-affine ELBs and autoscaling groups, single-IP ELBs, continuous improvements to EBS and S3, etc.) Given the scale of their infrastructure in us-east-1, it's now also highly unlikely that any customer will manage to overload it on their own.
  - Nick-Craver 12y ago
    I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :) They are getting much better, as all providers are. They're still just not a fit for many people because of performance requirements that are either impossible or too costly to meet on that type of infrastructure. I've always said this: the cloud isn't a good fit for us; do what works for you.
    - duggan 12y ago
      > I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :) If you're going to bring that up, I can't resist reminding you of that time you had poor sysadmins running up and down stairs with buckets of fuel to keep your servers running[1]. > I've always said this: the cloud isn't a good fit for us; do what works for you. There are costs. Frankly, it appears Stack Exchange prefers to lay those on its people rather than its purse. [1]: http://blog.stackoverflow.com/2012/11/se-podcast-36-we-got-hit-by-a-hurricane/ http://blog.stackoverflow.com/2012/11/se-podcast-36-we-got-h...
balls187 12y ago
Are you at the same scale as Stack Exchange?
- thethimble 12y ago
  That's hardly an argument. Netflix, consuming 33% of the nation's bandwidth is a great counterexample. In terms of scale, GoGuardian (the company I co-founded) has passed Stack Exchange. Articles like this make me so happy to be on AWS. Delegating this work to AWS allows us to focus on the product instead of the hardware that it runs on.
  - deleted 12y ago
    [deleted]
  - balls187 12y ago
    > That's hardly an argument. Netflix, consuming 33% of the nation's bandwidth is a great counterexample. Actually it isn't. Can AWS handle the scale of StackOverflow? Absolutely. Netflix is a GREAT example of the scale AWS can handle. However, can netflix migrate their entire system off of AWS without careful planning (which the parent of my comment was scoffing at)? I'm a huge proponent of AWS, but I wouldn't be so naive to say that AWS allows me to be oblivious of hardware and infrastructure costs. > In terms of scale, GoGuardian (the company I co-founded) has passed Stack Exchange. How so? StackOverflow is #56 in the world in Internet Traffic Rank (alexa.com).

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

by jmccree 12y ago

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

Nick-Craver 12y ago

The cloud is awesome if you have zero interest whatsoever in hardware. It's not without trade-offs (nor is the other direction), and too many for us - but if it works for you then great I say. We obviously feel very different, and are just doing what works best for us.

balls187 12y ago
> We obviously feel very different, and are just doing what works best for us. And doing it very well, if I may say so.
beachstartup 12y ago
most of the public cloud cheer-leading is just people rationalizing to themselves what an awesome decision they made deploying on aws onto a billion tiny instances or whatever. and for lots of folks, it probably is pretty awesome. however, i've seen very, very few people compare actual before/after $ figures on hn. when it comes time to show your cards everyone gets cold feet, either because they 1. don't have any idea because they aren't the ones paying for it or 2. don't have a baseline for comparison and are just paying whatever amazon asks or 3. it ended up costing 2-4x as much on amazon when your bill is $5k/month 2-4x isn't that big of a deal. when it's $100k+ a month, it becomes a really big deal.

rdoherty 12y ago

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a launch configuration and updating an auto-scaling group. Maybe an hour of my time to update configs, verify and deploy. Updating something like a database or caching servers is more work for sure, but with 0 time needed to get to the DC, unpack, rack and configure servers you do save time with 'the cloud'. I get that you do pay more for EC2 instances, especially if you keep hardware for 4 years. But AWS prices drop every year or two along with (generally) faster versions of software so your overall costs do drop. How many ops employees would you need for a fleet of 500 servers in a datacenter? We managed it all with 4 people with AWS.

bri3d 12y ago
The Stack Exchange philosophy is that because they can buy truly mega hardware (each one of those two blade chassis they bought has 72 cores and 1.4TB of RAM, remember!), they don't need those 500 servers to start with. Plus the hardware is an asset and you get to depreciate it. Everywhere I've ever worked we've had the "big spreadsheet" of projected cloud costs, projected ops costs, and hardware costs. In general the "scale horizontally" philosophy will favor the cloud while the "scale vertical" philosophy still seems to favor owned hardware in local datacenters. Which is superior is a crazy, long-standing debate with no clear answer.
- MrBuddyCasino 12y ago
  Can you elaborate? I thought the answer to that question was to scale up if you can, because its much simpler and therefore cheaper. Similar to how you don't give up ACID unless the scale you're working at doesn't permit it anymore.
  - thirsteh 12y ago
    > I thought the answer to that question was to scale up if you can, because its much simpler and therefore cheaper ✱cough✱ SQL Server licensing fees ✱cough✱
    - bri3d 12y ago
      That's a good point that I overlooked in my post - I'm sure this is a huge consideration for Stack Exchange. For what it's worth Basecamp also evangelize the "scale up" approach and they're on an open-source stack.
  - bri3d 12y ago
    There's never really a "one size fits all" answer, which is why it's a long-running debate and depends heavily on the product. Scaling horizontally can let you use smaller, cheaper hardware on average and burst to higher capacity more easily if you need to, at the expense of a lot of complexity. It also (done right, which is rare) tends to gain you a greater degree of fault tolerance, since hardware instances become rapidly-replaceable commodities. Most web apps have spiky but relatively predictable load. For example, a typical enterprise SaaS startup gets more traffic during work hours than on weekends. For these companies the complexity of developing a horizontally scaled architecture can be offset by the decreased cost of buying really big machines for peak load and then scaling back to a couple small instances for periods of below-average load. That's (ostensibly) why AWS exists in the first place: Amazon had to buy a lot of peak capacity for Black Friday and Christmas and found it going unused the rest of the year. They never meant to sell their excess capacity, but they realized the tools that they built to dynamically scale their infrastructure were valuable to others. Plus, a lot of work is offline data analytics, ETL, and so on. It's very cost effective to scale these workloads horizontally on-demand - spin up extra workers to run your reporting each hour/night and keep costs down the rest of the time when you don't need the capacity. On the flip side, companies like Stack Exchange and Basecamp have high, relatively stable traffic worldwide. For companies like this it makes more sense to scale vertically - if they were in the cloud, they would never scale down or shut down their instances anyway. Personally, I agree that horizontal scalability is oversold and most people can, indeed, scale up instead of out. However, a lot of plenty smart people disagree with me and have valid reasons to scale horizontally, too.

jimwalsh 12y ago

Actually, occasionally you do need to think about it. Reference the various times AWS emails customers about unplanned outages that need to occur because of hardware issues/patching/etc. Cloud is great, but there str still plenty of reasons to run your own datacenter. Yes for many startups it might not make sense, but at a certain sized company/application it can easily make sense.

ak217 12y ago
AWS just announced a few days ago that their latest Xen patch will be deployed through a live update to their hypervisor kernel, and that going forward they expect patches like this to be rolled out live. The real upside of AWS is that they have relentlessly pursued and killed off reasons for you to care about things like this. They've eliminated points of failure in their infrastructure and given operators a wealth of tools to ensure their apps stay up through any update or event (AZ-affine ELBs and autoscaling groups, single-IP ELBs, continuous improvements to EBS and S3, etc.) Given the scale of their infrastructure in us-east-1, it's now also highly unlikely that any customer will manage to overload it on their own.
- Nick-Craver 12y ago
  I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :) They are getting much better, as all providers are. They're still just not a fit for many people because of performance requirements that are either impossible or too costly to meet on that type of infrastructure. I've always said this: the cloud isn't a good fit for us; do what works for you.
  - duggan 12y ago
    > I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :) If you're going to bring that up, I can't resist reminding you of that time you had poor sysadmins running up and down stairs with buckets of fuel to keep your servers running[1]. > I've always said this: the cloud isn't a good fit for us; do what works for you. There are costs. Frankly, it appears Stack Exchange prefers to lay those on its people rather than its purse. [1]: http://blog.stackoverflow.com/2012/11/se-podcast-36-we-got-hit-by-a-hurricane/ http://blog.stackoverflow.com/2012/11/se-podcast-36-we-got-h...

balls187 12y ago

Are you at the same scale as Stack Exchange?

thethimble 12y ago
That's hardly an argument. Netflix, consuming 33% of the nation's bandwidth is a great counterexample. In terms of scale, GoGuardian (the company I co-founded) has passed Stack Exchange. Articles like this make me so happy to be on AWS. Delegating this work to AWS allows us to focus on the product instead of the hardware that it runs on.
- deleted 12y ago
  [deleted]
- balls187 12y ago
  > That's hardly an argument. Netflix, consuming 33% of the nation's bandwidth is a great counterexample. Actually it isn't. Can AWS handle the scale of StackOverflow? Absolutely. Netflix is a GREAT example of the scale AWS can handle. However, can netflix migrate their entire system off of AWS without careful planning (which the parent of my comment was scoffing at)? I'm a huge proponent of AWS, but I wouldn't be so naive to say that AWS allows me to be oblivious of hardware and infrastructure costs. > In terms of scale, GoGuardian (the company I co-founded) has passed Stack Exchange. How so? StackOverflow is #56 in the world in Internet Traffic Rank (alexa.com).