5 ms·

This is so important. I see so many people farting around in the problems of highly scalable distributed cloud systems with thousands of nodes, not realizing th

This is so important. I see so many people farting around in the problems of highly scalable distributed cloud systems with thousands of nodes, not realizing that single-digit QPS per node is insanity, why don't you look there first? Computers are fast.

colecut 6y ago
I think they do this because their priority is decentralization and not speed
- closeparen 6y ago
  You can achieve decentralization with a handful of nodes, and you can manage them using simple techniques without inviting the whole large-scale cluster scheduler problem. What I'm talking about is like 3,000 nodes to serve 15,000 QPS. That seems like a lot of QPS, and so it makes intuitive sense to people that they'd need a big distributed system for it, unless they have a sense of how fast computers actually are.
  - H8crilA 6y ago
    As this post shows - it really depends on what you're doing :). There are services that can handle 100k QPS on a single machine, no problemo. There are services that can handle as little as 10 QPS per machine. Those are rather extreme cases. A classic example of a "cheap" service is something like Stack Overflow [1] or Wikipedia: serving content that is mostly static; you just gotta configure your caches well. A classic example of an "expensive" service: a search engine, where so many queries cannot be cached (too rare), or are even completely unique (never seen before [2]). [1] https://stackexchange.com/performance https://stackexchange.com/performance [2] https://www.seroundtable.com/google-15-percent-queries-25730.html https://www.seroundtable.com/google-15-percent-queries-25730...
    - closeparen 6y ago
      Right - "those are rather extreme cases." So if you're looking at 5 QPS / node, you should be raising your eyebrows, not automatically reaching for an autoscaling group.
      - deleted 6y ago
        [deleted]
    - mdoms 6y ago
      Stack Overflow caches almost nothing. In fact they have just removed most of their remaining caching[1] with no performance hit. Because, surprise, pulling data from a DB and rendering HTML from it is fast, if you're careful about how you do it. The problem with many developers today is that they don't take this level of care. [1] https://twitter.com/Nick_Craver/status/1280494336673751044?s=19 https://twitter.com/Nick_Craver/status/1280494336673751044?s...
- dahfizz 6y ago
  I think people do it because they buy into the hype train of some cloud tech and don't really think about if they need it or not.
- yachtman 6y ago
  No they do this because horizontal scalability is more general. Once you cross the threshold of what your meganode can handle you have to rewrite your code from the bottom up
  - deleted 6y ago
    [deleted]
lemmsjid 6y ago
Having spent a lot of time in highly scalable distributed system land, I can perhaps give you few reasons: 1. Sometimes business needs require non-linear growth quickly. One day your perfectly optimized process now requires an n-squared algo across billions of records. Suddenly your one machine is tiny compared to what it used to be. 2. If you haven't scaled the workload horizontally, it can lead to a rewrite just to begin to distribute the load. When you hit the limits of a single machine, it is often a hard barrier that you cannot easily cross. 3. Distributed systems are distributing 3 main resources: memory, IO, and computation. IO especially quickly becomes a bottleneck on a single system, but memory is also quite thorny because memory management itself can be a bottleneck on a vertically scaled system. 4. People with distributed systems do obsess over single-node QPS. If it takes 5k nodes to do work and you can optimize down to 2k nodes, you are saving a lot of money! However, it isn't that simple and this is where being properly distributed gives you cost leverage. You might find that 5 highly scaled machines are more costly than 50 commodity machines, especially in the cloud ecosystems. 5. Finally, and probably to your point and the GP's points, you kind of have to go with the flow when it comes to the level of abstraction people are writing the code at. It takes increasingly specialized knowledge to optimize a process that will run well on a 100gb process (e.g. virtual machine garbage collection issues). You are knowingly sacrificing efficiency for the nice higher level abstractions. That said, I would emphasize that the abstractions become a smaller slice of the performance pie when you're dealing with algorithmic complexity. C won't magically make your Python algo O(1). I don't disagree with your main point btw. I think a well done processing / data pipeline should have distributed systems available and single-node computational scenarios available, because there is an undeniable complexity gain when you reach for a distributed system immediately.
- closeparen 6y ago
  I'd argue that throughput that bad usually means you are doing something stupid and easy to fix, like making a blocking call in your single-threaded server process, doing per request what you should have done at startup, not indexing for your query pattern, etc. It's not like you need multiple person-years of Brendan Gregg-level wizardry to serve QPS in the low hundreds from <= 10 boxes. It should just happen, or else you should have a clear understanding of what about your workload is so intense.
therealdrag0 6y ago
Who do you see doing that? The places I’ve worked (small/med companies) with lots of instances were always well utilized. People understood well the cost of infrastructure and put effort into minimizing it. But as features and services and customers grow, so do instances.
therealdrag0 6y ago
Sometimes node count is a function of code organization (micro services) not throughput. And when the instances are collocated (10 on a single host) it’s not like they’re not utilizing the machines firepower. Individual instances might have low throughout but some and in aggregate there could be lots more.

colecut 6y ago

I think they do this because their priority is decentralization and not speed

closeparen 6y ago
You can achieve decentralization with a handful of nodes, and you can manage them using simple techniques without inviting the whole large-scale cluster scheduler problem. What I'm talking about is like 3,000 nodes to serve 15,000 QPS. That seems like a lot of QPS, and so it makes intuitive sense to people that they'd need a big distributed system for it, unless they have a sense of how fast computers actually are.
- H8crilA 6y ago
  As this post shows - it really depends on what you're doing :). There are services that can handle 100k QPS on a single machine, no problemo. There are services that can handle as little as 10 QPS per machine. Those are rather extreme cases. A classic example of a "cheap" service is something like Stack Overflow [1] or Wikipedia: serving content that is mostly static; you just gotta configure your caches well. A classic example of an "expensive" service: a search engine, where so many queries cannot be cached (too rare), or are even completely unique (never seen before [2]). [1] https://stackexchange.com/performance https://stackexchange.com/performance [2] https://www.seroundtable.com/google-15-percent-queries-25730.html https://www.seroundtable.com/google-15-percent-queries-25730...
  - closeparen 6y ago
    Right - "those are rather extreme cases." So if you're looking at 5 QPS / node, you should be raising your eyebrows, not automatically reaching for an autoscaling group.
    - deleted 6y ago
      [deleted]
  - mdoms 6y ago
    Stack Overflow caches almost nothing. In fact they have just removed most of their remaining caching[1] with no performance hit. Because, surprise, pulling data from a DB and rendering HTML from it is fast, if you're careful about how you do it. The problem with many developers today is that they don't take this level of care. [1] https://twitter.com/Nick_Craver/status/1280494336673751044?s=19 https://twitter.com/Nick_Craver/status/1280494336673751044?s...
dahfizz 6y ago
I think people do it because they buy into the hype train of some cloud tech and don't really think about if they need it or not.
yachtman 6y ago
No they do this because horizontal scalability is more general. Once you cross the threshold of what your meganode can handle you have to rewrite your code from the bottom up
- deleted 6y ago
  [deleted]

lemmsjid 6y ago

Having spent a lot of time in highly scalable distributed system land, I can perhaps give you few reasons: 1. Sometimes business needs require non-linear growth quickly. One day your perfectly optimized process now requires an n-squared algo across billions of records. Suddenly your one machine is tiny compared to what it used to be. 2. If you haven't scaled the workload horizontally, it can lead to a rewrite just to begin to distribute the load. When you hit the limits of a single machine, it is often a hard barrier that you cannot easily cross. 3. Distributed systems are distributing 3 main resources: memory, IO, and computation. IO especially quickly becomes a bottleneck on a single system, but memory is also quite thorny because memory management itself can be a bottleneck on a vertically scaled system. 4. People with distributed systems do obsess over single-node QPS. If it takes 5k nodes to do work and you can optimize down to 2k nodes, you are saving a lot of money! However, it isn't that simple and this is where being properly distributed gives you cost leverage. You might find that 5 highly scaled machines are more costly than 50 commodity machines, especially in the cloud ecosystems. 5. Finally, and probably to your point and the GP's points, you kind of have to go with the flow when it comes to the level of abstraction people are writing the code at. It takes increasingly specialized knowledge to optimize a process that will run well on a 100gb process (e.g. virtual machine garbage collection issues). You are knowingly sacrificing efficiency for the nice higher level abstractions. That said, I would emphasize that the abstractions become a smaller slice of the performance pie when you're dealing with algorithmic complexity. C won't magically make your Python algo O(1). I don't disagree with your main point btw. I think a well done processing / data pipeline should have distributed systems available and single-node computational scenarios available, because there is an undeniable complexity gain when you reach for a distributed system immediately.

closeparen 6y ago
I'd argue that throughput that bad usually means you are doing something stupid and easy to fix, like making a blocking call in your single-threaded server process, doing per request what you should have done at startup, not indexing for your query pattern, etc. It's not like you need multiple person-years of Brendan Gregg-level wizardry to serve QPS in the low hundreds from <= 10 boxes. It should just happen, or else you should have a clear understanding of what about your workload is so intense.

therealdrag0 6y ago

Who do you see doing that? The places I’ve worked (small/med companies) with lots of instances were always well utilized. People understood well the cost of infrastructure and put effort into minimizing it. But as features and services and customers grow, so do instances.

therealdrag0 6y ago

Sometimes node count is a function of code organization (micro services) not throughput. And when the instances are collocated (10 on a single host) it’s not like they’re not utilizing the machines firepower. Individual instances might have low throughout but some and in aggregate there could be lots more.