4 ms·

Great bits of knowledge, though to throw something out there. In some circles when presented with the definition of high scale being a 1000+ plus. They will lik

by yaaaq 10y ago

Great bits of knowledge, though to throw something out there. In some circles when presented with the definition of high scale being a 1000+ plus. They will likely reply with; I forgot how to count that low. Life gets interesting at 10,000+ servers.

wjossey 10y ago
Life was plenty interesting at 1,000 (although we were plenty larger when you added in non user facing servers). :)
- ddorian43 10y ago
  Can I ask how big were these servers, and would it be ~nicer to have half x2 bigger servers ?
  - wjossey 10y ago
    Really depends on the role of the server. We didn't tend to have "monster" servers that were gigantic and served multiple purposes. We had "flavors" that matched workflow needs and were sizes to make sure we could maximally utilize our bottleneck resource. So, for example, our app servers typically were always scaled by CPU capacity. Thus, we tried to maximize the cost per compute unit (so if you were in Amazon this would correlate to one of their "c" flavors). Even if a fleet of servers was in the dozens or hundreds, we didn't usually try to use the largest flavor available. The reasoning behind this was mostly that making web apps and databases fully utilize massive machines can sometimes be a bit of a dark art. We repeatedly found a horizontally scaled fleet of "commodity" sized boxes was less of a headache than a small cluster of vertically scaled boxes. You'll notice in my post above that I favor horizontally scalable technologies at every layer, and it's for that reason. Our database instances went wide before they went tall. If we needed more memory or disk we added a server of the same type, which adds incremental capacity. In some cases, we had emergencies which necessitated a vertical scaling event, so it was always nice having that bullet to fire when needed. So, to wrap up my response to answer your question, no, not really. We didn't have any issues at 1000 that we really didn't also have at 500. Most of the servers we ran were no bigger or faster than what you could put in a small tower in your bedroom (mind you with Xeons instead of i7s). If I had to do it over again I'd stay middle of the pack as soon as I had enough traffic to fully balance across three servers and then scale from there.
sulam 10y ago
Feel free to elaborate how life gets interesting at larger scale, if you want to be constructive. Or not. The parent post gives a good template for how to be constructive, if you are inspired to take that path.

by yaaaq 10y ago

wjossey 10y ago

Life was plenty interesting at 1,000 (although we were plenty larger when you added in non user facing servers). :)

ddorian43 10y ago
Can I ask how big were these servers, and would it be ~nicer to have half x2 bigger servers ?
- wjossey 10y ago
  Really depends on the role of the server. We didn't tend to have "monster" servers that were gigantic and served multiple purposes. We had "flavors" that matched workflow needs and were sizes to make sure we could maximally utilize our bottleneck resource. So, for example, our app servers typically were always scaled by CPU capacity. Thus, we tried to maximize the cost per compute unit (so if you were in Amazon this would correlate to one of their "c" flavors). Even if a fleet of servers was in the dozens or hundreds, we didn't usually try to use the largest flavor available. The reasoning behind this was mostly that making web apps and databases fully utilize massive machines can sometimes be a bit of a dark art. We repeatedly found a horizontally scaled fleet of "commodity" sized boxes was less of a headache than a small cluster of vertically scaled boxes. You'll notice in my post above that I favor horizontally scalable technologies at every layer, and it's for that reason. Our database instances went wide before they went tall. If we needed more memory or disk we added a server of the same type, which adds incremental capacity. In some cases, we had emergencies which necessitated a vertical scaling event, so it was always nice having that bullet to fire when needed. So, to wrap up my response to answer your question, no, not really. We didn't have any issues at 1000 that we really didn't also have at 500. Most of the servers we ran were no bigger or faster than what you could put in a small tower in your bedroom (mind you with Xeons instead of i7s). If I had to do it over again I'd stay middle of the pack as soon as I had enough traffic to fully balance across three servers and then scale from there.

sulam 10y ago

Feel free to elaborate how life gets interesting at larger scale, if you want to be constructive. Or not. The parent post gives a good template for how to be constructive, if you are inspired to take that path.