7 ms·

How and why we switched to Amazon EC2

51 points by moritzplassnig 14y ago

127001brewer 14y ago
Linode was out of the picture fast, as they do not provide per hour pricing. I thought that Linode is competitively priced - is that a wrong assumption? (What is the pricing of Amazon's EC2 per year compared to Linode's?)
- matlock 14y ago
  Linode is very competitively priced, but we need to scale up/down on an hourly basis. when there are less builds to run we simply want less servers to run. Linode is much cheaper than Amazon when you go over the whole year, but it just doesn't fit our needs.
- sp332 14y ago
  Linode is cheaper than Amazon, but then every service that charges per-month (not per-hour) is cheaper than Amazon.
  - ajross 14y ago
    CPU time on Linode is cheaper than standard-priced EC2 instances. Compared to a reserved instance they lose badly, though. And last I checked, storage on Linode was 8x higher than EBS.
    - astrodust 14y ago
      Linode's storage is prohibitively expensive compared to EBS and S3. Their instance sizes are all based on a $1/GB flat rate with no volume discounts. If anything, this has to be slashed in half somehow. It's been a few years since this has been adjusted.
    - sp332 14y ago
      OK, I didn't remember storage being that bad. Also, reserved instances are billed per-year, so my per-hour comparison stands.
  - astrodust 14y ago
    Linode charges up-front for the month, but if you terminate an instance early a pro-rated amount is credited to your account. The granularity is at least a day.
  - aristidb 14y ago
    Did you consider Reserved Instances?
    - dirkk0 14y ago
      As far as I am concerned, Reserved Instances are the way to go, if you consider to get to be be a heavy user. I opened a spreadsheet recently on my own startup idea, and with Reserved Instances Amazon EC2 got very competitive.
      - matlock 14y ago
        Reserved Instances are the next thing we are looking into as well to get our costs down dramatically
- ajitk 14y ago
  Recently stumbled upon very competitively priced DigitalOcean: https://www.digitalocean.com/price-comparison-chart https://www.digitalocean.com/price-comparison-chart They have APIs to manage on-demand instances as well.
  - matlock 14y ago
    Digital Ocean looks pretty sweet, but the advantages by going with EC2 for us are a large number of different Instance Types we can use for every task we have. It provides us with a lot of flexibility on how to build our service.
    - ajitk 14y ago
      That is true. Amazon has been in cloud hosting space for a long time to cover almost all areas required to run a server. I am rooting for competitors though who could offer more services. That should bring interesting services and drive down cost even further.
  - adanto6840 14y ago
    Thank you for the link! Their offering looks pretty solid; they are apparently using SSDs for all disk space which is one of the first I've seen doing this while using an 'on-demand' pricing model that's competitive with Linode. Some key things I saw when I briefly checked it out: -Their answers [1] regarding bandwidth bursting and fair bandwidth usage / sharing policies [2] is slightly ambiguous, or worrisome at least IMO, although maybe I'm just a cynic heh. -They do include CPU bursting [3] similar to Linode. -No private IP addresses [4]. No mention of IPv6 either but perhaps they do support it? =( -Their 'Features' page looks great but I can't find more specific information & details on some of the things mentioned such as: A) US & Europe data center; is casually mentioned as being New York & Amsterdam only though. B) The ability to upgrade a 'Droplet' VM to a dedicated instance that can use all the host resources. C) No mention of VM resize speed / more info on the 'private network' aspects. [1] = https://www.digitalocean.com/community/questions/do-you-limit-bandwidth https://www.digitalocean.com/community/questions/do-you-limi... [2] = https://www.digitalocean.com/community/questions/do-you-have-a-fair-usage-policy-for-bandwidth https://www.digitalocean.com/community/questions/do-you-have... [3] = https://www.digitalocean.com/community/questions/how-does-cpu-scheduling-work https://www.digitalocean.com/community/questions/how-does-cp... [4] = https://www.digitalocean.com/community/questions/do-you-offer-private-ips https://www.digitalocean.com/community/questions/do-you-offe... Does anyone have any first-hand experience with DigitalOcean that could provide some insight? I'll probably give it a whirl in the next few days as it looks to be a solid offering; would definitely be interested if anyone that may have gone with them already though!
- jimktrains2 14y ago
  They'll refund unused time.
ezequiel-garzon 14y ago
What networking issues did you experience with Hetzner?
- matlock 14y ago
  Builds failed from time to time because the network was simply gone. We worked around the issues by running commands again when they timed out, but it was one of the reasons (the main reason being that having your own servers is simply not flexible enough) for switching from Hetzner
  - ashayh 14y ago
    Network partitioning is also a common occurrence on AWS, and people have to plan for that.
    - matlock 14y ago
      Of course, but it happened with a regularity that just didn't work out for us any more. Although it was definitely not the main issue to change. So far though we haven't seen any of those problems on EC2 (or at least only very very very few instances)
mvkel 14y ago
How do you get around the IO issue? We've been looking at EC2 for, well, forever, but IO performance has always been a huge issue -- to the point where it'd cost more than hosting it ourselves just to scale IO performance to the point where it'd equal a couple self-hosted servers.
- ajross 14y ago
  Not all services are I/O bound. Hell, it's routine in the modern world to have datasets for real world products that fit in, what, $20 of RAM?
  - matlock 14y ago
    Which is exactly the case in our environment. As soon as all libraries are loaded from EBS IO is not that much of an issue any more
- matlock 14y ago
  We store all the code and everything that changes in a Ram disk, so really fast. The virtual machine lies on EBS, but it is really fast to start and performance is not limited by EBS at all in our case.
- killface 14y ago
  when i last used ec2, it was almost exclusively for async backend processes. in those cases, high throughput is important, but latency is not a factor in the same way. If there's a 300ms delay talking to the system from the outside world, it's no big deal. that said, netflix seems to have solved the IO issue with ec2. I'd be curious to know more about their findings.
  - jedberg 14y ago
    Main way we solve IO issues: We don't rely on EBS -- most data is stored on the local drives using Cassandra. For cache throughput, our cache library adds zone affinity to memcache, so the clients are hitting the cache in the same zone first. Also, we're now exploring their SSD options (http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html http://techblog.netflix.com/2012/07/benchmarking-high-perfor...)
peterwwillis 14y ago
If the main motivator here is price (ability to charge per hour vs per month), and EC2 is so much more expensive than a VPS provider, have you done the math to see if buying a couple extra VPS's for the month would not break even or be cheaper overall? Linode is kind of pricey compared to other VPS providers. The larger your site gets the less you're going to need to rely on "hourly" scaling (as your traffic models also become more predictable). There's also the question of uptime, which we've all seen is not a guarantee from just using Amazon.
- grey-area 14y ago
  From the link: As a hosted continuous deployment service releasing changes and scaling automatically is a necessity. I think the key here is that this blog is talking about a hosted service which responds on demand to their client's needs on an hourly basis. EC2 has a lot of infrastructure which does this for them (as long as they can automate it) and is specifically built for this, whereas they'd have to write it themselves by interfacing with a VPS API otherwise. So EC2 might be worth the money to them. VPS providers are mostly more focussed on giving you 1 or 2 boxes to use, and then you leave your site running with occasional adjustments in capacity etc, not on spinning up and down servers on demand, which is a bit more specialised, and might require tens or even hundreds of servers. For a normal site that's not going to be relevant, and a VPS is cheaper, for this site, it seems EC2 is a better fit...
  - matlock 14y ago
    Exactly. Price itself is not the main issue, but flexibility is. For example on Weekends we typically see 20%-30% the traffic that we see during the week, simply because people are not at work. Scaling up and down is an absolute necessity there.
    - bluestix 14y ago
      Dedicated servers billed monthly cost 15% of what Amazon instances cost. So you would still be better off with dedicated hardware.
  - 14y ago
matlock 14y ago
As I was asked about the implementation details I'll write a second blogpost at the end of the week that goes into the details of our implementation.
huoju 14y ago
we plan to switch to AWS too. Amazon has so many services, you need to learn how to combine them. May be a new job title will become hot in the further: AWS Architect.
mp99e99 14y ago
Dont forget www.atlantic.net/cloud fast io competitive price per sec billing API etc

Linode was out of the picture fast, as they do not provide per hour pricing. I thought that Linode is competitively priced - is that a wrong assumption? (What is the pricing of Amazon's EC2 per year compared to Linode's?)

matlock 14y ago
Linode is very competitively priced, but we need to scale up/down on an hourly basis. when there are less builds to run we simply want less servers to run. Linode is much cheaper than Amazon when you go over the whole year, but it just doesn't fit our needs.
sp332 14y ago
Linode is cheaper than Amazon, but then every service that charges per-month (not per-hour) is cheaper than Amazon.
- ajross 14y ago
  CPU time on Linode is cheaper than standard-priced EC2 instances. Compared to a reserved instance they lose badly, though. And last I checked, storage on Linode was 8x higher than EBS.
  - astrodust 14y ago
    Linode's storage is prohibitively expensive compared to EBS and S3. Their instance sizes are all based on a $1/GB flat rate with no volume discounts. If anything, this has to be slashed in half somehow. It's been a few years since this has been adjusted.
  - sp332 14y ago
    OK, I didn't remember storage being that bad. Also, reserved instances are billed per-year, so my per-hour comparison stands.
- astrodust 14y ago
  Linode charges up-front for the month, but if you terminate an instance early a pro-rated amount is credited to your account. The granularity is at least a day.
- aristidb 14y ago
  Did you consider Reserved Instances?
  - dirkk0 14y ago
    As far as I am concerned, Reserved Instances are the way to go, if you consider to get to be be a heavy user. I opened a spreadsheet recently on my own startup idea, and with Reserved Instances Amazon EC2 got very competitive.
    - matlock 14y ago
      Reserved Instances are the next thing we are looking into as well to get our costs down dramatically
ajitk 14y ago
Recently stumbled upon very competitively priced DigitalOcean: https://www.digitalocean.com/price-comparison-chart https://www.digitalocean.com/price-comparison-chart They have APIs to manage on-demand instances as well.
- matlock 14y ago
  Digital Ocean looks pretty sweet, but the advantages by going with EC2 for us are a large number of different Instance Types we can use for every task we have. It provides us with a lot of flexibility on how to build our service.
  - ajitk 14y ago
    That is true. Amazon has been in cloud hosting space for a long time to cover almost all areas required to run a server. I am rooting for competitors though who could offer more services. That should bring interesting services and drive down cost even further.
- adanto6840 14y ago
  Thank you for the link! Their offering looks pretty solid; they are apparently using SSDs for all disk space which is one of the first I've seen doing this while using an 'on-demand' pricing model that's competitive with Linode. Some key things I saw when I briefly checked it out: -Their answers [1] regarding bandwidth bursting and fair bandwidth usage / sharing policies [2] is slightly ambiguous, or worrisome at least IMO, although maybe I'm just a cynic heh. -They do include CPU bursting [3] similar to Linode. -No private IP addresses [4]. No mention of IPv6 either but perhaps they do support it? =( -Their 'Features' page looks great but I can't find more specific information & details on some of the things mentioned such as: A) US & Europe data center; is casually mentioned as being New York & Amsterdam only though. B) The ability to upgrade a 'Droplet' VM to a dedicated instance that can use all the host resources. C) No mention of VM resize speed / more info on the 'private network' aspects. [1] = https://www.digitalocean.com/community/questions/do-you-limit-bandwidth https://www.digitalocean.com/community/questions/do-you-limi... [2] = https://www.digitalocean.com/community/questions/do-you-have-a-fair-usage-policy-for-bandwidth https://www.digitalocean.com/community/questions/do-you-have... [3] = https://www.digitalocean.com/community/questions/how-does-cpu-scheduling-work https://www.digitalocean.com/community/questions/how-does-cp... [4] = https://www.digitalocean.com/community/questions/do-you-offer-private-ips https://www.digitalocean.com/community/questions/do-you-offe... Does anyone have any first-hand experience with DigitalOcean that could provide some insight? I'll probably give it a whirl in the next few days as it looks to be a solid offering; would definitely be interested if anyone that may have gone with them already though!
jimktrains2 14y ago
They'll refund unused time.

ezequiel-garzon 14y ago

What networking issues did you experience with Hetzner?

matlock 14y ago
Builds failed from time to time because the network was simply gone. We worked around the issues by running commands again when they timed out, but it was one of the reasons (the main reason being that having your own servers is simply not flexible enough) for switching from Hetzner
- ashayh 14y ago
  Network partitioning is also a common occurrence on AWS, and people have to plan for that.
  - matlock 14y ago
    Of course, but it happened with a regularity that just didn't work out for us any more. Although it was definitely not the main issue to change. So far though we haven't seen any of those problems on EC2 (or at least only very very very few instances)

mvkel 14y ago

How do you get around the IO issue? We've been looking at EC2 for, well, forever, but IO performance has always been a huge issue -- to the point where it'd cost more than hosting it ourselves just to scale IO performance to the point where it'd equal a couple self-hosted servers.

ajross 14y ago
Not all services are I/O bound. Hell, it's routine in the modern world to have datasets for real world products that fit in, what, $20 of RAM?
- matlock 14y ago
  Which is exactly the case in our environment. As soon as all libraries are loaded from EBS IO is not that much of an issue any more
matlock 14y ago
We store all the code and everything that changes in a Ram disk, so really fast. The virtual machine lies on EBS, but it is really fast to start and performance is not limited by EBS at all in our case.
killface 14y ago
when i last used ec2, it was almost exclusively for async backend processes. in those cases, high throughput is important, but latency is not a factor in the same way. If there's a 300ms delay talking to the system from the outside world, it's no big deal. that said, netflix seems to have solved the IO issue with ec2. I'd be curious to know more about their findings.
- jedberg 14y ago
  Main way we solve IO issues: We don't rely on EBS -- most data is stored on the local drives using Cassandra. For cache throughput, our cache library adds zone affinity to memcache, so the clients are hitting the cache in the same zone first. Also, we're now exploring their SSD options (http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html http://techblog.netflix.com/2012/07/benchmarking-high-perfor...)

peterwwillis 14y ago

If the main motivator here is price (ability to charge per hour vs per month), and EC2 is so much more expensive than a VPS provider, have you done the math to see if buying a couple extra VPS's for the month would not break even or be cheaper overall? Linode is kind of pricey compared to other VPS providers. The larger your site gets the less you're going to need to rely on "hourly" scaling (as your traffic models also become more predictable). There's also the question of uptime, which we've all seen is not a guarantee from just using Amazon.

grey-area 14y ago
From the link: As a hosted continuous deployment service releasing changes and scaling automatically is a necessity. I think the key here is that this blog is talking about a hosted service which responds on demand to their client's needs on an hourly basis. EC2 has a lot of infrastructure which does this for them (as long as they can automate it) and is specifically built for this, whereas they'd have to write it themselves by interfacing with a VPS API otherwise. So EC2 might be worth the money to them. VPS providers are mostly more focussed on giving you 1 or 2 boxes to use, and then you leave your site running with occasional adjustments in capacity etc, not on spinning up and down servers on demand, which is a bit more specialised, and might require tens or even hundreds of servers. For a normal site that's not going to be relevant, and a VPS is cheaper, for this site, it seems EC2 is a better fit...
- matlock 14y ago
  Exactly. Price itself is not the main issue, but flexibility is. For example on Weekends we typically see 20%-30% the traffic that we see during the week, simply because people are not at work. Scaling up and down is an absolute necessity there.
  - bluestix 14y ago
    Dedicated servers billed monthly cost 15% of what Amazon instances cost. So you would still be better off with dedicated hardware.
- 14y ago

matlock 14y ago

As I was asked about the implementation details I'll write a second blogpost at the end of the week that goes into the details of our implementation.

huoju 14y ago

we plan to switch to AWS too. Amazon has so many services, you need to learn how to combine them. May be a new job title will become hot in the further: AWS Architect.

mp99e99 14y ago

Dont forget www.atlantic.net/cloud fast io competitive price per sec billing API etc