6 ms·

"With Amazon we experienced networking issues, hanging VM instances, unpredictable performance degradation (probably due to noisy neighbors sharing our hardware

"With Amazon we experienced networking issues, hanging VM instances, unpredictable performance degradation (probably due to noisy neighbors sharing our hardware, but there was no way to know) and numerous other problems. " Why do I get the feeling it was kind of a cop-out to just pack up and move without finding the root cause? I've seen it plenty of times: the "best solution" is to just find a different hosting provider. In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code.

toomuchtodo 12y ago
> In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code. You haven't been using Amazon long enough then. Amazon is great for proof of concept. No upfront costs, extremely scalable, etc. Unfortunately, its expensive compared to physical hardware once you get to scale, and you may never solve underlying performance issues due to it being a shared tenant environment, even if you're a Netflix-sized customer.
- josh2600 12y ago
  You can use abstraction layers to isolate yourself from issues with the underlying metal. For example, I had a good thread the last time the maintenance reboots happened: https://news.ycombinator.com/item?id=9120289 https://news.ycombinator.com/item?id=9120289. Solving multi-tenancy issues is hard, but not impossible. I think it's a lot easier with live migration. If a box is giving you problems, just move the load to a new box while maintaining the same IP addressing. With respect to cost, yes, AWS gets expensive at scale, but if you're at scale your servers are generally not your major cost center (it's usually payroll and licensing).
  - deitcher 12y ago
    That is what I like about the cloud. Running in your own hardware lets you be, well, "lazy" about application architecture. Running in a place where it is shared and instances can disappear forces you to design a lot more robustly and nimbly. Of course, that design discipline is great wherever you are running....
    - toomuchtodo 12y ago
      Hardware is almost always cheaper than engineering time. Don't optimize prematurely.
      - deitcher 12y ago
        Yes, but engineering is cheaper than running after real-time problems.
        toomuchtodo 12y ago
        Now we get to the technical debt debate. Sometimes, you have to make good decisions now instead of perfect decisions later. The market doesn't care how elegant your code is.
        josh2600 12y ago
        The market cares about uptime, and cost relative to return. Perfect should never be the enemy of the good.
        toomuchtodo 12y ago
        > Perfect should never be the enemy of the good. Only in a world where resources are infinite does this work.
        deitcher 12y ago
        No debate, I agree with everything you have said. But sometimes you have to clean that garbage up because it does matter to the market.
- TheMagicHorsey 12y ago
  But doesn't Netflix successfully use AWS now? Its possible they get special treatment if they are big enough (nobody else's jobs on their physical machines ... or something like that).
  - Thaxll 12y ago
    Netflix uses thousands of instances, they don't share servers.
    - babo 12y ago
      Using thousands of servers doesn't mean that you don't share physical servers.
      - deleted 12y ago
        [deleted]
      - babo 12y ago
        Even if you own all the VMs on a physical server it's hard to avoid the noisy neighbour problem. Netflix has its own instance monitoring tool - Vector - to handle these issues: http://techblog.netflix.com/2015/02/a-microscope-on-microservices.html http://techblog.netflix.com/2015/02/a-microscope-on-microser...
  - kovyrin 12y ago
    Yep, that is why I specifically noted in the article, that given enough resources, it is possible to survive and even thrive in the cloud (Netflix being one of the best examples of that), but in case of a startup it is not aways the best idea to keep burning money and engineering resources when your primary job is to keep building a business. And no, I don't think they're getting any real special treatment from Amazon since every single talk from a Netflix engineer points out the the cloud-specific issues they're solving in their infrastructure software.
adrianpike 12y ago
I've seen and fought issues with hanging and/or stalling EC2 instances, and the decision was made to move to physical hardware - it was one of those tradeoff choices you have to make about paying N dollars to throw money at hardware vs. M dollars at person-hours to investigate. It's definitely kicking the can down the road (eventually you have to build such that failing infrastructure is transparent to your eng team), but I still think it was the right decision at the time. YMMV obviously. :)
- makeitsuckless 12y ago
  I've had the odd problem with hanging/stalling instances before I learned to design for the cloud. Now those instances automatically get replaced with new ones and nobody would even notice if I didn't pipe the notice through to Slack. Of course, this only works if whatever you use it for allows for this. However, in most cases not using EC2 as a stateless throwaway computing resources is simply a matter of bad infrastructure design.

toomuchtodo 12y ago

> In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code. You haven't been using Amazon long enough then. Amazon is great for proof of concept. No upfront costs, extremely scalable, etc. Unfortunately, its expensive compared to physical hardware once you get to scale, and you may never solve underlying performance issues due to it being a shared tenant environment, even if you're a Netflix-sized customer.

josh2600 12y ago
You can use abstraction layers to isolate yourself from issues with the underlying metal. For example, I had a good thread the last time the maintenance reboots happened: https://news.ycombinator.com/item?id=9120289 https://news.ycombinator.com/item?id=9120289. Solving multi-tenancy issues is hard, but not impossible. I think it's a lot easier with live migration. If a box is giving you problems, just move the load to a new box while maintaining the same IP addressing. With respect to cost, yes, AWS gets expensive at scale, but if you're at scale your servers are generally not your major cost center (it's usually payroll and licensing).
- deitcher 12y ago
  That is what I like about the cloud. Running in your own hardware lets you be, well, "lazy" about application architecture. Running in a place where it is shared and instances can disappear forces you to design a lot more robustly and nimbly. Of course, that design discipline is great wherever you are running....
  - toomuchtodo 12y ago
    Hardware is almost always cheaper than engineering time. Don't optimize prematurely.
    - deitcher 12y ago
      Yes, but engineering is cheaper than running after real-time problems.
      - toomuchtodo 12y ago
        Now we get to the technical debt debate. Sometimes, you have to make good decisions now instead of perfect decisions later. The market doesn't care how elegant your code is.
        josh2600 12y ago
        The market cares about uptime, and cost relative to return. Perfect should never be the enemy of the good.
        toomuchtodo 12y ago
        > Perfect should never be the enemy of the good. Only in a world where resources are infinite does this work.
        deitcher 12y ago
        No debate, I agree with everything you have said. But sometimes you have to clean that garbage up because it does matter to the market.
TheMagicHorsey 12y ago
But doesn't Netflix successfully use AWS now? Its possible they get special treatment if they are big enough (nobody else's jobs on their physical machines ... or something like that).
- Thaxll 12y ago
  Netflix uses thousands of instances, they don't share servers.
  - babo 12y ago
    Using thousands of servers doesn't mean that you don't share physical servers.
    - deleted 12y ago
      [deleted]
    - babo 12y ago
      Even if you own all the VMs on a physical server it's hard to avoid the noisy neighbour problem. Netflix has its own instance monitoring tool - Vector - to handle these issues: http://techblog.netflix.com/2015/02/a-microscope-on-microservices.html http://techblog.netflix.com/2015/02/a-microscope-on-microser...
- kovyrin 12y ago
  Yep, that is why I specifically noted in the article, that given enough resources, it is possible to survive and even thrive in the cloud (Netflix being one of the best examples of that), but in case of a startup it is not aways the best idea to keep burning money and engineering resources when your primary job is to keep building a business. And no, I don't think they're getting any real special treatment from Amazon since every single talk from a Netflix engineer points out the the cloud-specific issues they're solving in their infrastructure software.

adrianpike 12y ago

I've seen and fought issues with hanging and/or stalling EC2 instances, and the decision was made to move to physical hardware - it was one of those tradeoff choices you have to make about paying N dollars to throw money at hardware vs. M dollars at person-hours to investigate. It's definitely kicking the can down the road (eventually you have to build such that failing infrastructure is transparent to your eng team), but I still think it was the right decision at the time. YMMV obviously. :)

makeitsuckless 12y ago
I've had the odd problem with hanging/stalling instances before I learned to design for the cloud. Now those instances automatically get replaced with new ones and nobody would even notice if I didn't pipe the notice through to Slack. Of course, this only works if whatever you use it for allows for this. However, in most cases not using EC2 as a stateless throwaway computing resources is simply a matter of bad infrastructure design.