5 ms·

As someone who has worked at high scale (1,000+ load balanced servers), this is a nice "primer" I'd share with a new(er) engineer, or someone who has not worked

by wjossey 10y ago

As someone who has worked at high scale (1,000+ load balanced servers), this is a nice "primer" I'd share with a new(er) engineer, or someone who has not worked in a horizontally scaled environment. A few things I'll point out in addition to this article: [1] I've found I greatly prefer databases that handle the resiliency and horizontal scalability natively (or natively in their respective client libraries). DBs natively scaling has been one of the greatest advancements in web technologies during my career, and I'd really recommend taking a look at the technologies like Riak & Cassandra for inspiration. [2] When scaling horizontally with load balancing, connection pooling becomes something to always pay attention to. If you have 1,000 servers, and 100 database instances, you likely don't want 100,000 active connections. Focusing on things like intra-zone preferences for servers (Server A prefers all DBs in its own rack, "zone", or "region") can be a real headache saver as you scale up. [3] Capacity planning is always key. What are you trying to plan for? Do you want to survive an entire AWS zone outage? Do you want to be able to survive a region outage? Do you worry about single servers going down, and if something "bigger" happens you'll just eat the downtime? Make sure you and your business stakeholders agree on what "resiliency" looks like to your business. It'll help you balance cost and downtime appropriately. [4] Remember when you're looking at spare capacity that you take into account the situation where a cache layer suffers an outage. If you have to fall back to the primary source of data, and it potentially is also suffering from degradation of service, what sort of capacity will you have?

0xbadcafebee 10y ago
There needs to be more publications covering this, because I don't see this becoming common knowledge or practiced very often. Then there's the modern design decisions that make some of these subjects taboo, like "you should never cache anything" and "a single node can handle a million concurrent connections so why worry" and "let's make all our machines use 99% of their capacity with containers because it's more efficient and hope we can deploy standbys before the database queues too much and blows the entire stack up". With the emergence of the pseudo-Devops cargo cult, people who like using JavaScript for server backend code are designing infrastructure they've never administered before. (sorry for the rant) There was a really great paper published a few years ago that was basically an Everything You Need To Know To Run Large Scale Enterprise Systems type paper. I can never find it when these types of threads come up, but it was dozens of pages long and really in-depth.
- torinmr 10y ago
  The SRE Book (http://shop.oreilly.com/product/0636920041528.do http://shop.oreilly.com/product/0636920041528.do) has a number of chapters on load balancing that are well worth reading. One thing that the book covers that I think this article glossed over was the fact that in sufficiently large systems there's never a single "load balancer" - instead there's many layers of load balancing systems at different levels of the stack. E.g.: DNS load balancing -> high capacity network-level load balancing -> shared reverse HTTP proxy -> application server -> database (with a "load balancer" internal to the application server load balancing among DB replicas).
  - peterwwillis 10y ago
    They mention there are different applications for load balancers, and then use one or two examples. But their main points are the most important ones: if you don't know what you're using it for, it might be worthless. The SRE book seems to only have a couple sub-chapters on it, so I would recommend looking up white papers and best practices for your particular application of load balancing. Safari Books has over 17,000 matches for "load balancing" in its library, for example, and F5 has 12 white papers on it just for their products.
- noam87 10y ago
  replying just in case someone finds what paper that was.
- theptip 10y ago
  http://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf http://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf ? This was featured on Adrian Colyer's blog earlier this year (https://blog.acolyer.org/2016/09/12/on-designing-and-deploying-internet-scale-services/ https://blog.acolyer.org/2016/09/12/on-designing-and-deployi...)
yaaaq 10y ago
Great bits of knowledge, though to throw something out there. In some circles when presented with the definition of high scale being a 1000+ plus. They will likely reply with; I forgot how to count that low. Life gets interesting at 10,000+ servers.
- wjossey 10y ago
  Life was plenty interesting at 1,000 (although we were plenty larger when you added in non user facing servers). :)
  - ddorian43 10y ago
    Can I ask how big were these servers, and would it be ~nicer to have half x2 bigger servers ?
    - wjossey 10y ago
      Really depends on the role of the server. We didn't tend to have "monster" servers that were gigantic and served multiple purposes. We had "flavors" that matched workflow needs and were sizes to make sure we could maximally utilize our bottleneck resource. So, for example, our app servers typically were always scaled by CPU capacity. Thus, we tried to maximize the cost per compute unit (so if you were in Amazon this would correlate to one of their "c" flavors). Even if a fleet of servers was in the dozens or hundreds, we didn't usually try to use the largest flavor available. The reasoning behind this was mostly that making web apps and databases fully utilize massive machines can sometimes be a bit of a dark art. We repeatedly found a horizontally scaled fleet of "commodity" sized boxes was less of a headache than a small cluster of vertically scaled boxes. You'll notice in my post above that I favor horizontally scalable technologies at every layer, and it's for that reason. Our database instances went wide before they went tall. If we needed more memory or disk we added a server of the same type, which adds incremental capacity. In some cases, we had emergencies which necessitated a vertical scaling event, so it was always nice having that bullet to fire when needed. So, to wrap up my response to answer your question, no, not really. We didn't have any issues at 1000 that we really didn't also have at 500. Most of the servers we ran were no bigger or faster than what you could put in a small tower in your bedroom (mind you with Xeons instead of i7s). If I had to do it over again I'd stay middle of the pack as soon as I had enough traffic to fully balance across three servers and then scale from there.
sasas 10y ago
Thanks for the contribution. What is the preferred approach for geographical redundancy in an active/standby site configuration? F5 offer BIG-IP GTM. I'm wondering if there are good open source equivalents?
- mnutt 10y ago
  What sort of availability guarantees do you need? If you can tolerate a minute or two of less-than-full availability, Route53 DNS using geographical failover is a decent solution.
  - sasas 10y ago
    Looking at four 9s (4.38 min a month). Solution must be self hosted and not reliant on commercial cloud infrastructure.
    - siculars 10y ago
      I'm at basho makers of Riak. If you're looking for ha at the db take a look.
      - sasas 10y ago
        Thanks. Not looking at HA with DB. I took a look at Riak briefly - looks interesting.
    - wjossey 10y ago
      I haven't been in a situation where we had geographic redundancy and that small an SLA, and a commercial gear requirement. Ive got a couple of ideas, but most of them likely violate one of your requirements. The best advice I can give is to see if you can get someone from a large company that self hosts to chat with you. Not too many companies have the sort of requirements you're looking at, so it's going to be of the ilk of someone at Microsoft, Google, FB, Oracle, etc. Search your network, see if you've got a connection to someone on an SRE team there, and ask to buy them a coffee. :)

by wjossey 10y ago

0xbadcafebee 10y ago

There needs to be more publications covering this, because I don't see this becoming common knowledge or practiced very often. Then there's the modern design decisions that make some of these subjects taboo, like "you should never cache anything" and "a single node can handle a million concurrent connections so why worry" and "let's make all our machines use 99% of their capacity with containers because it's more efficient and hope we can deploy standbys before the database queues too much and blows the entire stack up". With the emergence of the pseudo-Devops cargo cult, people who like using JavaScript for server backend code are designing infrastructure they've never administered before. (sorry for the rant) There was a really great paper published a few years ago that was basically an Everything You Need To Know To Run Large Scale Enterprise Systems type paper. I can never find it when these types of threads come up, but it was dozens of pages long and really in-depth.

torinmr 10y ago
The SRE Book (http://shop.oreilly.com/product/0636920041528.do http://shop.oreilly.com/product/0636920041528.do) has a number of chapters on load balancing that are well worth reading. One thing that the book covers that I think this article glossed over was the fact that in sufficiently large systems there's never a single "load balancer" - instead there's many layers of load balancing systems at different levels of the stack. E.g.: DNS load balancing -> high capacity network-level load balancing -> shared reverse HTTP proxy -> application server -> database (with a "load balancer" internal to the application server load balancing among DB replicas).
- peterwwillis 10y ago
  They mention there are different applications for load balancers, and then use one or two examples. But their main points are the most important ones: if you don't know what you're using it for, it might be worthless. The SRE book seems to only have a couple sub-chapters on it, so I would recommend looking up white papers and best practices for your particular application of load balancing. Safari Books has over 17,000 matches for "load balancing" in its library, for example, and F5 has 12 white papers on it just for their products.
noam87 10y ago
replying just in case someone finds what paper that was.
theptip 10y ago
http://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf http://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf ? This was featured on Adrian Colyer's blog earlier this year (https://blog.acolyer.org/2016/09/12/on-designing-and-deploying-internet-scale-services/ https://blog.acolyer.org/2016/09/12/on-designing-and-deployi...)

yaaaq 10y ago

Great bits of knowledge, though to throw something out there. In some circles when presented with the definition of high scale being a 1000+ plus. They will likely reply with; I forgot how to count that low. Life gets interesting at 10,000+ servers.

wjossey 10y ago
Life was plenty interesting at 1,000 (although we were plenty larger when you added in non user facing servers). :)
- ddorian43 10y ago
  Can I ask how big were these servers, and would it be ~nicer to have half x2 bigger servers ?
  - wjossey 10y ago
    Really depends on the role of the server. We didn't tend to have "monster" servers that were gigantic and served multiple purposes. We had "flavors" that matched workflow needs and were sizes to make sure we could maximally utilize our bottleneck resource. So, for example, our app servers typically were always scaled by CPU capacity. Thus, we tried to maximize the cost per compute unit (so if you were in Amazon this would correlate to one of their "c" flavors). Even if a fleet of servers was in the dozens or hundreds, we didn't usually try to use the largest flavor available. The reasoning behind this was mostly that making web apps and databases fully utilize massive machines can sometimes be a bit of a dark art. We repeatedly found a horizontally scaled fleet of "commodity" sized boxes was less of a headache than a small cluster of vertically scaled boxes. You'll notice in my post above that I favor horizontally scalable technologies at every layer, and it's for that reason. Our database instances went wide before they went tall. If we needed more memory or disk we added a server of the same type, which adds incremental capacity. In some cases, we had emergencies which necessitated a vertical scaling event, so it was always nice having that bullet to fire when needed. So, to wrap up my response to answer your question, no, not really. We didn't have any issues at 1000 that we really didn't also have at 500. Most of the servers we ran were no bigger or faster than what you could put in a small tower in your bedroom (mind you with Xeons instead of i7s). If I had to do it over again I'd stay middle of the pack as soon as I had enough traffic to fully balance across three servers and then scale from there.

sasas 10y ago

Thanks for the contribution. What is the preferred approach for geographical redundancy in an active/standby site configuration? F5 offer BIG-IP GTM. I'm wondering if there are good open source equivalents?

mnutt 10y ago
What sort of availability guarantees do you need? If you can tolerate a minute or two of less-than-full availability, Route53 DNS using geographical failover is a decent solution.
- sasas 10y ago
  Looking at four 9s (4.38 min a month). Solution must be self hosted and not reliant on commercial cloud infrastructure.
  - siculars 10y ago
    I'm at basho makers of Riak. If you're looking for ha at the db take a look.
    - sasas 10y ago
      Thanks. Not looking at HA with DB. I took a look at Riak briefly - looks interesting.
  - wjossey 10y ago
    I haven't been in a situation where we had geographic redundancy and that small an SLA, and a commercial gear requirement. Ive got a couple of ideas, but most of them likely violate one of your requirements. The best advice I can give is to see if you can get someone from a large company that self hosts to chat with you. Not too many companies have the sort of requirements you're looking at, so it's going to be of the ilk of someone at Microsoft, Google, FB, Oracle, etc. Search your network, see if you've got a connection to someone on an SRE team there, and ask to buy them a coffee. :)