5 ms·

Not every problem is network or io bound. Imagine a system that ingests data from clients. Only a subset of the capacity may service client connections, while t

Not every problem is network or io bound. Imagine a system that ingests data from clients. Only a subset of the capacity may service client connections, while the rest may be running computations over data.

javajosh 4y ago
Yes, of course you might be CPU bound, in which case even a single user might take up 50k servers (think of a scientist doing a climate simulation, or an AI researcher building a very large model). All things being equal, CPU-bound is the exception, not the rule. Most every program we think of as an "application" is chat with some structure, persistence, access control added, and are indeed IO bound.
foobiekr 4y ago
Basically in an ideal system you are only bound by hardware limits. memory bandwidth bound memory size bound cache hit rate and bandwidth bound TLB size bound CPU decode and issue logic bound CPU renaming / OOO buffer bound kernel implementation bound (esp. locks, interrupts,...) network physical layer bound GPU/TPU bound (which are variations of the above, though mostly memory) power and cooling bound ... ... so basically unless you're running out of functional units, you're hopefully I/O bound depending on where you consider i/o. But in reality, most SW is just "crappy SW making poor use of resources bound." That's where we mostly are now as an industry. Bad language choices, terrible design, no cross-layer comprehension.
- javajosh 4y ago
  >But in reality, most SW is just "crappy SW making poor use of resources bound." I agree, but the number one culprit is premature distribution, thanks to the widely pervasive cargo culting Amazon-style microservices.
  - aserafini 4y ago
    I often think about this. We have 100s of services running on 1000s of servers and 10,000s of containers but I wonder how much of that is ‘waste’ in the sense that we are running 10,000x Operating Systems, 100s of load balancers, database per service/region etc. Seems possible that horizontal scaling at every layer can be wasteful, but the alternatives are hard for me to even conceive.
    - javajosh 4y ago
      Regarding container processes, it depends on how your containers are written. In the best case, each process is not so much an OS as a kernel instance - a kind of poor man's unikernel. It's an extremely stateless process that requires the bare minimum of the physical host. The overhead that concerns me more is messaging. A network of N nodes has N! paths. In the general case it doesn't take long before the overhead of internal messaging absolutely dominates all other processing in the steady-state. Most people take a brute force approach of partitioning the network on purpose, or even intentionally bottle-necking (again) all the traffic. What's really hilarious is when you see people architecting microservices with kafka and apogee with all the uservice trimmings, only to deploy everything to a single physical rack, or even a single beefy machine. I feel like the proper path to scalability goes through repeated breakage, because then you get to see how things actually break under load, and what can be done to fix it. For example, you can do a lot by moving logic to stored procs, or equivalently, moving your db into your application process. But people seem to think these are non-starters, for some reason, probably because the notion of stateless app servers as the key to horizontal scalability has become a Law, even though there are alternatives. But good luck questioning foundational assumptions - the risk is just too damn high.
- legulere 4y ago
  Having enough computing power to be able to write crappy software is an enormous productivity boost. Writing software with crappy performance is way easier than writing software with good performance and takes less time and less experienced developers. Writing crappy software is often better at achieving business goals as cheap as possible.
- DSingularity 4y ago
  “Cross layer comprehension” - what is that? Wanna move computation to the NIC ?
  - foobiekr 4y ago
    It is useful to understand the underlying layers so as to take advantage of what they are good at and not end up in the position of relying on operations where they perform badly. "What is are the natural, performant operations for the layer below me? How can I construct my solution from these operations instead of pretending the layers are orthogonal?" A good example would be the linear read/write performance of harddisks. If you had the option to avoid random access and take advantage of read-ahead and other methods you'd have seen much better performance than an approach that ignored this behavior. There are many, many examples of this.
  - astrange 4y ago
    That is a good idea and we do that. (TSO/LRO/TLS offload)
- jiggawatts 4y ago
  There was a post (here?) about how Netflix uses NICs that offload TLS encryption because that’s literally the only way to hit 200 Gbps. It’s not that the CPUs can’t encrypt that fast — the limit is the memory bandwidth. I see some clouds deploying servers with 100 Gbps NICs and I wonder what percentage of deployed applications could get anywhere near that…

javajosh 4y ago

Yes, of course you might be CPU bound, in which case even a single user might take up 50k servers (think of a scientist doing a climate simulation, or an AI researcher building a very large model). All things being equal, CPU-bound is the exception, not the rule. Most every program we think of as an "application" is chat with some structure, persistence, access control added, and are indeed IO bound.

foobiekr 4y ago

Basically in an ideal system you are only bound by hardware limits. memory bandwidth bound memory size bound cache hit rate and bandwidth bound TLB size bound CPU decode and issue logic bound CPU renaming / OOO buffer bound kernel implementation bound (esp. locks, interrupts,...) network physical layer bound GPU/TPU bound (which are variations of the above, though mostly memory) power and cooling bound ... ... so basically unless you're running out of functional units, you're hopefully I/O bound depending on where you consider i/o. But in reality, most SW is just "crappy SW making poor use of resources bound." That's where we mostly are now as an industry. Bad language choices, terrible design, no cross-layer comprehension.

javajosh 4y ago
>But in reality, most SW is just "crappy SW making poor use of resources bound." I agree, but the number one culprit is premature distribution, thanks to the widely pervasive cargo culting Amazon-style microservices.
- aserafini 4y ago
  I often think about this. We have 100s of services running on 1000s of servers and 10,000s of containers but I wonder how much of that is ‘waste’ in the sense that we are running 10,000x Operating Systems, 100s of load balancers, database per service/region etc. Seems possible that horizontal scaling at every layer can be wasteful, but the alternatives are hard for me to even conceive.
  - javajosh 4y ago
    Regarding container processes, it depends on how your containers are written. In the best case, each process is not so much an OS as a kernel instance - a kind of poor man's unikernel. It's an extremely stateless process that requires the bare minimum of the physical host. The overhead that concerns me more is messaging. A network of N nodes has N! paths. In the general case it doesn't take long before the overhead of internal messaging absolutely dominates all other processing in the steady-state. Most people take a brute force approach of partitioning the network on purpose, or even intentionally bottle-necking (again) all the traffic. What's really hilarious is when you see people architecting microservices with kafka and apogee with all the uservice trimmings, only to deploy everything to a single physical rack, or even a single beefy machine. I feel like the proper path to scalability goes through repeated breakage, because then you get to see how things actually break under load, and what can be done to fix it. For example, you can do a lot by moving logic to stored procs, or equivalently, moving your db into your application process. But people seem to think these are non-starters, for some reason, probably because the notion of stateless app servers as the key to horizontal scalability has become a Law, even though there are alternatives. But good luck questioning foundational assumptions - the risk is just too damn high.
legulere 4y ago
Having enough computing power to be able to write crappy software is an enormous productivity boost. Writing software with crappy performance is way easier than writing software with good performance and takes less time and less experienced developers. Writing crappy software is often better at achieving business goals as cheap as possible.
DSingularity 4y ago
“Cross layer comprehension” - what is that? Wanna move computation to the NIC ?
- foobiekr 4y ago
  It is useful to understand the underlying layers so as to take advantage of what they are good at and not end up in the position of relying on operations where they perform badly. "What is are the natural, performant operations for the layer below me? How can I construct my solution from these operations instead of pretending the layers are orthogonal?" A good example would be the linear read/write performance of harddisks. If you had the option to avoid random access and take advantage of read-ahead and other methods you'd have seen much better performance than an approach that ignored this behavior. There are many, many examples of this.
- astrange 4y ago
  That is a good idea and we do that. (TSO/LRO/TLS offload)
jiggawatts 4y ago
There was a post (here?) about how Netflix uses NICs that offload TLS encryption because that’s literally the only way to hit 200 Gbps. It’s not that the CPUs can’t encrypt that fast — the limit is the memory bandwidth. I see some clouds deploying servers with 100 Gbps NICs and I wonder what percentage of deployed applications could get anywhere near that…