6 ms·

I'm Ryan Park -- I'm the Pinterest engineer quoted in the article. I'm happy to answer any questions about our setup. Just to clarify, the auto-scaling was spe

I'm Ryan Park -- I'm the Pinterest engineer quoted in the article. I'm happy to answer any questions about our setup. Just to clarify, the auto-scaling was specifically for a pool of web application servers. At the time I gathered the numbers, there were 80 servers in that pool. In the last few months we've been moving toward a service-oriented architecture, and we've been able to use the same code to auto-scale the internal services. Of course it's not possible to auto-scale stateful servers like databases, but it's still saving us a considerable amount of money. We implemented the auto-scaling in early 2012, so it's been in use for almost a year now. It only took about 2 weeks of engineering to build the system. It does need occasional maintenance, but it's still worth the effort given how much money it saves us.

ntoshev 14y ago
How do you decide whether to run a spot or on-demand instance to handle dynamic load? What happens if spot instance cost suddenly spikes? Also, how do you use regions/availability zones?
- RyanGWU82 14y ago
  Right now we run everything in the US-East region, and we have all our services balanced across 4 availability zones. If there's a problem in a single AZ, it will affect every layer of our system, but only about 25% of the hosts in that layer. Some of our services are automatically resilient and can handle that easily. Others aren't so great, but we're working on more automatic failover. When we need more servers for an auto-scaled service, we open spot requests and start on-demand instances at the same time. For most services, we want to run about 50% on-demand and 50% spot. We have a watchdog process that continually checks what's running. It launches more instances whenever there aren't enough, and terminates instances when there are too many. So if the spot price spikes and a bunch of our spot instances are shut down, the watchdog will launch replacement instances on-demand. It will also request more spot instances once the price has dropped back to normal. In reality we don't often run into spot capacity issues -- maybe once a month, and it's almost never apparent to our users. I spoke about this in detail at AWS re:Invent last month, and the full talk is available online here: http://www.youtube.com/watch?v=73-G2zQ9sHU http://www.youtube.com/watch?v=73-G2zQ9sHU
achompas 14y ago
Thanks! How do your watchdog instances monitor other hosts? Do they track usage/load via snmpd, or something else? Also, are they directly calling the EC2 API to launch or shut down hosts, or do you have an in-house deployment system?
- RyanGWU82 14y ago
  Our watchdog process calls the EC2 APIs directly to identify how many instances are running, which ones are spot instances, etc. Boto, the AWS client library for Python, makes that pretty easy. The watchdog isn't very sophisticated -- it just checks to make sure that the correct number of instances are running in each auto-scale group. Our application servers aren't very efficient in certain respects, so we don't trust metrics like usage/load to make auto-scaling decisions. If I was doing it over again, I'd just use Amazon's auto-scaling features for all of this. At the time we built this, EC2's auto-scaling didn't support some of the features we needed. Since then, they've made it a lot easier to do things like set up a repeating schedule for auto-scaling, rather than using metrics. We only have one EC2 AMI that we use for all of our servers. That AMI is pretty basic; it only does enough to connect to our Puppet configuration management servers. Puppet then configures the boxes as web servers (or databases, or...) and adds them to the appropriate load balancer.
  - achompas 14y ago
    Very interesting, thanks! I've spent a bit of time working on a library to accomplish this, and Boto has been extremely helpful. The watchdog isn't very sophisticated -- it just checks to make sure that the correct number of instances are running in each auto-scale group. How did you decide on the right number of instances? The article mentions 20%--is that based on latency, a cost-saving target, or something else?
    - RyanGWU82 14y ago
      We revise the "right number of instances" every few weeks based on latency and traffic numbers. But sometimes when we release updates, we'll find that we suddenly need a lot more capacity (or a lot less if we improved performance). We have automated tools to help us notice performance regressions. Once we decide that we need to change the pool size, we adjust the watchdog configuration by hand.
aioprisan 14y ago
it would be nice if you could share some of the logic that the watchdog uses in figuring out when hosts are shut down and others fired up
jedberg 14y ago
> Of course it's not possible to auto-scale stateful servers like databases That's not entirely true. :) It's just a lot harder to scale them elastically. At Netflix we've done some proof of concept work on elastically scaling Cassandra, although we don't have it in production yet. I think that is one of our goals for 2013.
- thezilch 14y ago
  Theoretically, there's going to be a state of affairs where it's not possible -- that's physics. Maybe with multi-host tenant, so that we're not relying on AWS networking / control-plane screwing with what is going to be the most important part of that dance. But this is about saving money; not being a tenant of multiple providers; not spending more than "2 weeks."
  - stonemetal 14y ago
    Even on a boring old DB: Spin up a few read slaves, give them a few min to sync to current, turn them loose. Sure it is only read scalability, but every bit counts.
    - thezilch 14y ago
      I think you're missing the point that I'm not going after the DB layer or your own software states (eg. read only). Most AWS outages are related to network layers; you have to already be sync'd. If you want to do writes, sync'd enough to still guarantee a quorum for your writes AND reads, and if the AWS network(s) are fractured enough, you could theoretically be without access to bringing up a stable "wan" for yourself. And then you may simply be hamstrung by ELB outages and be left without hope altogether. If we're talking about read-only, I don't see why a DB, in the traditional meaning, has much to do with the issue. These are not necessarily AWS faults; they are certainly trying to help. It might be the year for being a tenant of more than one PaaS though, but that is only going to up your costs and still not give infinite 9s.
- hnriot 14y ago
  Id be happy enough if you could just keep Netflix running more often.
ericabiz 14y ago
Why would you not choose to go colo and save the money? Even with hiring an extra full-time person, going colo should drop your costs by at least half vs. Amazon.
- RyanGWU82 14y ago
  Our #1 requirement has been to keep up with the growth in traffic on the site. We've been growing so fast that there's literally no way we could have ordered and racked equipment fast enough. We were also a very small team -- a year ago there were only about a dozen people in the whole company. At this point we're much larger, which gives us room to consider more options like colo or multiple cloud providers. AWS certainly feels pretty costly when you compare colo prices to the list price for on-demand instances. But one of the reasons I wanted to present our work is to show that you can use the cloud for a lot less than the list price. It takes work to buy reserved instances or run spot instances, but that does make it much more cost competitive.
- samstave 14y ago
  I am not sure what you mean exactly. By "going colo" they would need to layout all the upfront hardware costs. These would not be insignificant. They would then have all the operational overhead of maintaining all that gear. You may have a higher monthly cost for AWS services - but without needing to buy any physical hardware, you have way less to worry about. Further - if you need to scale, it can be done in seconds rather than weeks/months given lead times for procurement, design time, implementation time (i.e. scheduling the install in the colo, coordinating the need for more space etc...) This is just scratching the surface...
  - vidarh 14y ago
    > By "going colo" they would need to layout all the upfront hardware costs. Leasing still ends up substantially cheaper than EC2. Never paid upfront for any colocated hardware I've been responsible for. So does managed hosting at a number of providers. Last time I priced out this, EC2 ended up 2-3 times as expensive as managed hosting (with no upfront costs for the managed hosting either), and the gap to leasing servers and putting them in a colo was even larger (but there you do need some scale before you cover the extra ops costs). > They would then have all the operational overhead of maintaining all that gear. If you're small enough, sure, your savings won't pay for extra ops people. But you don't need to be very large before the savings outweigh the cost of more ops people. And with managed hosting this is a non-issue - at that point you don't have any more ops issues than you have with EC2. > Further - if you need to scale, it can be done in seconds rather than weeks/months given lead times for procurement, design time, implementation time (i.e. scheduling the install in the colo, coordinating the need for more space etc...) It's not either/or. In fact, being prepared to use EC2 to handle peaks means the cost difference between self/colo hosted (+occasional EC2 use for peaks) and EC2 gets even larger, as you can run your own servers at far closer to full capacity without the risk you'd take if you didn't have that ability. Handling occasional peaks with EC2 is a great use of it, and definitively cost effective. > Further - if you need to scale, it can be done in seconds rather than weeks/months given lead times for procurement, design time, implementation time (i.e. scheduling the install in the colo, coordinating the need for more space etc...) See above. But also consider that if instead comparing against the managed hosting option, a number of providers with auto-provision in minutes to a couple of hours once an order is placed. And many providers now also offer a mix between colo, managed hosting and EC2 like cloud solutions, so you can put your base load in a rented rack, scale in the mid term via managed hosting, and spin up cloud instances as needed if you want to deal with a single provider. EC2 is great for "quick and dirty" temporary solutions, batch jobs or handling peaks that last less than about 6-8 hours a day, and I use it now and again for that reason. But the moment your instances are up more than about 8 hours a day, and you have more than a few of them, it will quickly start costing you more than the alternatives.
ck2 14y ago
Please make a way to authenticate incoming pinterest EC2 requests from the web vs other garbage traffic from EC2. Or just simply publish your outgoing EC2 ip pool list. We cannot completely block EC2 because of Pinterest and that's a bad situation.
- kawsper 14y ago
  Why do you need to block EC2?
  - jacquesm 14y ago
    Because it is a source of some very bad traffic. Anybody doing web-facing stuff that is targeted by bots/jerks/spammers/all of the above will block hosting facilities wholesale.
    - Negitivefrags 14y ago
      I am interested in doing this. Do you have any information on how to find out which IP ranges belong to server hosts? I couldn't find any useful results searching google.
      - jacquesm 14y ago
        here are a few lists to get you started: http://proxy-ip-list.com/download/proxy-list-port-3128.txt http://proxy-ip-list.com/download/proxy-list-port-3128.txt http://proxy-ip-list.com/download/free-usa-proxy-ip.txt http://proxy-ip-list.com/download/free-usa-proxy-ip.txt http://www.proxylists.net/http_highanon.txt http://www.proxylists.net/http_highanon.txt http://www.proxylists.net/socks4.txt http://www.proxylists.net/socks4.txt http://www.proxylists.net/socks5.txt http://www.proxylists.net/socks5.txt http://www.stopforumspam.com/downloads/listed_ip_90.zip http://www.stopforumspam.com/downloads/listed_ip_90.zip I've got my own db of hosting facilities which I made by taking 100M urls and doing a lookup on the hostname, then saving the IP found in a db. This gives you some level of confidence that a certain class 'C' is used for hosting.
pavs 14y ago
Can we have some stats? Number of Servers? How much bandwidth per month? How much space needed and at what rate is it growing? Traffic stats (if possible)?
mcgwiz 14y ago
Do you use any Light or Medium Reserved? Or are they all Heavy?
trustfundbaby 14y ago
I'm interested in your db setup, specifically how do you replicate and locate them (regionally) to handle requests from the application instances, So that a request from Spain goes to the right server instead of trying to hit a db server all the way in Texas ... for example.
jcastro 14y ago
Out of curiosity what OS do you use? And do you make your own custom AMIs for various workloads?

ntoshev 14y ago

How do you decide whether to run a spot or on-demand instance to handle dynamic load? What happens if spot instance cost suddenly spikes? Also, how do you use regions/availability zones?

RyanGWU82 14y ago
Right now we run everything in the US-East region, and we have all our services balanced across 4 availability zones. If there's a problem in a single AZ, it will affect every layer of our system, but only about 25% of the hosts in that layer. Some of our services are automatically resilient and can handle that easily. Others aren't so great, but we're working on more automatic failover. When we need more servers for an auto-scaled service, we open spot requests and start on-demand instances at the same time. For most services, we want to run about 50% on-demand and 50% spot. We have a watchdog process that continually checks what's running. It launches more instances whenever there aren't enough, and terminates instances when there are too many. So if the spot price spikes and a bunch of our spot instances are shut down, the watchdog will launch replacement instances on-demand. It will also request more spot instances once the price has dropped back to normal. In reality we don't often run into spot capacity issues -- maybe once a month, and it's almost never apparent to our users. I spoke about this in detail at AWS re:Invent last month, and the full talk is available online here: http://www.youtube.com/watch?v=73-G2zQ9sHU http://www.youtube.com/watch?v=73-G2zQ9sHU

achompas 14y ago

Thanks! How do your watchdog instances monitor other hosts? Do they track usage/load via snmpd, or something else? Also, are they directly calling the EC2 API to launch or shut down hosts, or do you have an in-house deployment system?

RyanGWU82 14y ago
Our watchdog process calls the EC2 APIs directly to identify how many instances are running, which ones are spot instances, etc. Boto, the AWS client library for Python, makes that pretty easy. The watchdog isn't very sophisticated -- it just checks to make sure that the correct number of instances are running in each auto-scale group. Our application servers aren't very efficient in certain respects, so we don't trust metrics like usage/load to make auto-scaling decisions. If I was doing it over again, I'd just use Amazon's auto-scaling features for all of this. At the time we built this, EC2's auto-scaling didn't support some of the features we needed. Since then, they've made it a lot easier to do things like set up a repeating schedule for auto-scaling, rather than using metrics. We only have one EC2 AMI that we use for all of our servers. That AMI is pretty basic; it only does enough to connect to our Puppet configuration management servers. Puppet then configures the boxes as web servers (or databases, or...) and adds them to the appropriate load balancer.
- achompas 14y ago
  Very interesting, thanks! I've spent a bit of time working on a library to accomplish this, and Boto has been extremely helpful. The watchdog isn't very sophisticated -- it just checks to make sure that the correct number of instances are running in each auto-scale group. How did you decide on the right number of instances? The article mentions 20%--is that based on latency, a cost-saving target, or something else?
  - RyanGWU82 14y ago
    We revise the "right number of instances" every few weeks based on latency and traffic numbers. But sometimes when we release updates, we'll find that we suddenly need a lot more capacity (or a lot less if we improved performance). We have automated tools to help us notice performance regressions. Once we decide that we need to change the pool size, we adjust the watchdog configuration by hand.

aioprisan 14y ago

it would be nice if you could share some of the logic that the watchdog uses in figuring out when hosts are shut down and others fired up

jedberg 14y ago

> Of course it's not possible to auto-scale stateful servers like databases That's not entirely true. :) It's just a lot harder to scale them elastically. At Netflix we've done some proof of concept work on elastically scaling Cassandra, although we don't have it in production yet. I think that is one of our goals for 2013.

thezilch 14y ago
Theoretically, there's going to be a state of affairs where it's not possible -- that's physics. Maybe with multi-host tenant, so that we're not relying on AWS networking / control-plane screwing with what is going to be the most important part of that dance. But this is about saving money; not being a tenant of multiple providers; not spending more than "2 weeks."
- stonemetal 14y ago
  Even on a boring old DB: Spin up a few read slaves, give them a few min to sync to current, turn them loose. Sure it is only read scalability, but every bit counts.
  - thezilch 14y ago
    I think you're missing the point that I'm not going after the DB layer or your own software states (eg. read only). Most AWS outages are related to network layers; you have to already be sync'd. If you want to do writes, sync'd enough to still guarantee a quorum for your writes AND reads, and if the AWS network(s) are fractured enough, you could theoretically be without access to bringing up a stable "wan" for yourself. And then you may simply be hamstrung by ELB outages and be left without hope altogether. If we're talking about read-only, I don't see why a DB, in the traditional meaning, has much to do with the issue. These are not necessarily AWS faults; they are certainly trying to help. It might be the year for being a tenant of more than one PaaS though, but that is only going to up your costs and still not give infinite 9s.
hnriot 14y ago
Id be happy enough if you could just keep Netflix running more often.

ericabiz 14y ago

Why would you not choose to go colo and save the money? Even with hiring an extra full-time person, going colo should drop your costs by at least half vs. Amazon.

RyanGWU82 14y ago
Our #1 requirement has been to keep up with the growth in traffic on the site. We've been growing so fast that there's literally no way we could have ordered and racked equipment fast enough. We were also a very small team -- a year ago there were only about a dozen people in the whole company. At this point we're much larger, which gives us room to consider more options like colo or multiple cloud providers. AWS certainly feels pretty costly when you compare colo prices to the list price for on-demand instances. But one of the reasons I wanted to present our work is to show that you can use the cloud for a lot less than the list price. It takes work to buy reserved instances or run spot instances, but that does make it much more cost competitive.
samstave 14y ago
I am not sure what you mean exactly. By "going colo" they would need to layout all the upfront hardware costs. These would not be insignificant. They would then have all the operational overhead of maintaining all that gear. You may have a higher monthly cost for AWS services - but without needing to buy any physical hardware, you have way less to worry about. Further - if you need to scale, it can be done in seconds rather than weeks/months given lead times for procurement, design time, implementation time (i.e. scheduling the install in the colo, coordinating the need for more space etc...) This is just scratching the surface...
- vidarh 14y ago
  > By "going colo" they would need to layout all the upfront hardware costs. Leasing still ends up substantially cheaper than EC2. Never paid upfront for any colocated hardware I've been responsible for. So does managed hosting at a number of providers. Last time I priced out this, EC2 ended up 2-3 times as expensive as managed hosting (with no upfront costs for the managed hosting either), and the gap to leasing servers and putting them in a colo was even larger (but there you do need some scale before you cover the extra ops costs). > They would then have all the operational overhead of maintaining all that gear. If you're small enough, sure, your savings won't pay for extra ops people. But you don't need to be very large before the savings outweigh the cost of more ops people. And with managed hosting this is a non-issue - at that point you don't have any more ops issues than you have with EC2. > Further - if you need to scale, it can be done in seconds rather than weeks/months given lead times for procurement, design time, implementation time (i.e. scheduling the install in the colo, coordinating the need for more space etc...) It's not either/or. In fact, being prepared to use EC2 to handle peaks means the cost difference between self/colo hosted (+occasional EC2 use for peaks) and EC2 gets even larger, as you can run your own servers at far closer to full capacity without the risk you'd take if you didn't have that ability. Handling occasional peaks with EC2 is a great use of it, and definitively cost effective. > Further - if you need to scale, it can be done in seconds rather than weeks/months given lead times for procurement, design time, implementation time (i.e. scheduling the install in the colo, coordinating the need for more space etc...) See above. But also consider that if instead comparing against the managed hosting option, a number of providers with auto-provision in minutes to a couple of hours once an order is placed. And many providers now also offer a mix between colo, managed hosting and EC2 like cloud solutions, so you can put your base load in a rented rack, scale in the mid term via managed hosting, and spin up cloud instances as needed if you want to deal with a single provider. EC2 is great for "quick and dirty" temporary solutions, batch jobs or handling peaks that last less than about 6-8 hours a day, and I use it now and again for that reason. But the moment your instances are up more than about 8 hours a day, and you have more than a few of them, it will quickly start costing you more than the alternatives.

ck2 14y ago

Please make a way to authenticate incoming pinterest EC2 requests from the web vs other garbage traffic from EC2. Or just simply publish your outgoing EC2 ip pool list. We cannot completely block EC2 because of Pinterest and that's a bad situation.

kawsper 14y ago
Why do you need to block EC2?
- jacquesm 14y ago
  Because it is a source of some very bad traffic. Anybody doing web-facing stuff that is targeted by bots/jerks/spammers/all of the above will block hosting facilities wholesale.
  - Negitivefrags 14y ago
    I am interested in doing this. Do you have any information on how to find out which IP ranges belong to server hosts? I couldn't find any useful results searching google.
    - jacquesm 14y ago
      here are a few lists to get you started: http://proxy-ip-list.com/download/proxy-list-port-3128.txt http://proxy-ip-list.com/download/proxy-list-port-3128.txt http://proxy-ip-list.com/download/free-usa-proxy-ip.txt http://proxy-ip-list.com/download/free-usa-proxy-ip.txt http://www.proxylists.net/http_highanon.txt http://www.proxylists.net/http_highanon.txt http://www.proxylists.net/socks4.txt http://www.proxylists.net/socks4.txt http://www.proxylists.net/socks5.txt http://www.proxylists.net/socks5.txt http://www.stopforumspam.com/downloads/listed_ip_90.zip http://www.stopforumspam.com/downloads/listed_ip_90.zip I've got my own db of hosting facilities which I made by taking 100M urls and doing a lookup on the hostname, then saving the IP found in a db. This gives you some level of confidence that a certain class 'C' is used for hosting.

pavs 14y ago

Can we have some stats? Number of Servers? How much bandwidth per month? How much space needed and at what rate is it growing? Traffic stats (if possible)?

mcgwiz 14y ago

Do you use any Light or Medium Reserved? Or are they all Heavy?

trustfundbaby 14y ago

I'm interested in your db setup, specifically how do you replicate and locate them (regionally) to handle requests from the application instances, So that a request from Spain goes to the right server instead of trying to hit a db server all the way in Texas ... for example.

jcastro 14y ago

Out of curiosity what OS do you use? And do you make your own custom AMIs for various workloads?