5 ms·

I've worked on several projects where performance was an afterthought. After the product scaled a bit, it suddenly became the highest priority - but at that tim

by Genbox 4y ago

I've worked on several projects where performance was an afterthought. After the product scaled a bit, it suddenly became the highest priority - but at that time, it was impossible to fix. At least for everyone that created the problem to begin with. I've taught high performance data structures to dev teams. I've tried to explain how a complex problem can sometimes be solved with a simple algorithm. I've spent decades on attempting to show coworkers that applying a little comp-sci can have a profound effect in the end. But no. Unfortunately, it always fails. The mindset is always "making it work" and problem solving is brute-forcing the problem until it works. It takes a special kind of mindset to keep systems efficient. It is like painting a picture, but most seem to prefer doing it with a paint roller.

wizofaus 4y ago
And I've worked on systems where months were essentially squandered on performance improvements that never paid off because we never grew the customer base sufficiently for them to be worth while... I'm all for dedicating time and effort towards producing performant code, but it does come at a cost - in some cases, a cost of maintainability (for an extreme example there's always https://users.cs.utah.edu/~elb/folklore/mel.html https://users.cs.utah.edu/~elb/folklore/mel.html). In fact I'd suggest in general if you design a library of functions where obviousness/clarity/ease-of-use are your primary criteria, performance is likely to suffer. And there are undoubtedly cases where the cost of higher-grade hardware (in terms of speed and storage capacity) is vastly lower than that of more efficient software. I'd also say performance tuning quite often involves significant trade-offs that lead to much higher memory usage - caching may well be the only way to achieve significant gains at certain scales, but then as you scale up even further, the memory requirements of the caching start to become an issue in themselves. If there were a simple solution it would have been found by now.
- Genbox 4y ago
  Performance is not the same as efficiency, and efficiency can't be solved with more hardware. Let's say I build a sorting algorithm that is O(N^2) complexity and works fine for small inputs (takes <1 millisecond), but it is going to be used for large data systems. Suddenly it takes hundreds of thousands of hours to sort the data. One of the corps I worked with went full scalability in their architecture. One-click deployments, dynamic scaling of servers, rebalancing of databases, automatic provisioning of storage. They were handling 40-50k requests pr. second with their 15-ish large server farm, which could sale down to 5 servers, or up to 50-ish before it began to wobble. I got called in because the company had gotten a large client that needed 100k requests pr. second. They tried scaling the system to fit the need, but the whole thing got unstable and their solution was "more operations people to manage it". I built a custom solution for the backend. Took about two months. The new system could do about 2100k requests pr. second on one server. Scalability of the new system was ~90% efficient as well, so lots of capacity for the future. None of their developers understood computers or the science behind them. They were all educated and experienced developers, but none of that were applied to the problem. They were just assembling parts from the hardware store until something worked, and the resulting Frankenstein's Monster was put into production.
  - wizofaus 4y ago
    I'm struggling to believe any single server could usefully service 2100k (well over 2 million!) requests per second. Even Google, with their vast farm of servers, reportedly only process less than 100k requests per second globally. I've certainly read of servers capable of handling in the order of 1000k requests per second as a benchmark, but the requests are usually pretty trivial (the one I saw literally did no input processing at all, and just returned a single fixed byte! But was written in Java, surprisingly.) At any rate, I would think a tiny % of real-life systems actually need to be able to support that sort of load, and bringing in somebody to do the scalability work once it's clear it's needed seems like exactly the right strategy to me.
    - Genbox 4y ago
      Not serving, but handling 2100k requests. Your skepticism is rightly placed, as the HTTP protocol is yet an example of an inefficient protocol that nonetheless is used as the primary protocol on the internet. Some webservers[1] can serve millions of requests pr. second, but I'd never use HTTP in code where efficiency is key. No, I'm talking about handling requests. In this particular case, requests (32 to 64 bytes) were flowing through several services (on the same computer). I replaced the processing chain with a single application to remove the overhead of serialization between processes. Requests were filtered early in the pipeline, which made a ~55% reduction in the work needed. Requests were then batched into succinct data structures and processed via SIMD. Output used to be JSON, but I instead wrote a custom memory allocator and just memcpy the entire blob on to the wire. Before: No pre-filtering, off-the-shelf databases (PSQL), queue system for I/O, named pipes and TCP/IP for local data transfer. Lots of concurrency issues, thread starvation and I/O bound work. After: Agnessive pre-filtering, succinct data structures for cache coherence, no serialization overhead, SIMD processing. Can saturate a 32 core CPU with almost no overhead. [1] https://www.techempower.com/benchmarks/#section=data-r13&hw=ph&test=plaintext https://www.techempower.com/benchmarks/#section=data-r13&hw=...

by Genbox 4y ago

wizofaus 4y ago

And I've worked on systems where months were essentially squandered on performance improvements that never paid off because we never grew the customer base sufficiently for them to be worth while... I'm all for dedicating time and effort towards producing performant code, but it does come at a cost - in some cases, a cost of maintainability (for an extreme example there's always https://users.cs.utah.edu/~elb/folklore/mel.html https://users.cs.utah.edu/~elb/folklore/mel.html). In fact I'd suggest in general if you design a library of functions where obviousness/clarity/ease-of-use are your primary criteria, performance is likely to suffer. And there are undoubtedly cases where the cost of higher-grade hardware (in terms of speed and storage capacity) is vastly lower than that of more efficient software. I'd also say performance tuning quite often involves significant trade-offs that lead to much higher memory usage - caching may well be the only way to achieve significant gains at certain scales, but then as you scale up even further, the memory requirements of the caching start to become an issue in themselves. If there were a simple solution it would have been found by now.

Genbox 4y ago
Performance is not the same as efficiency, and efficiency can't be solved with more hardware. Let's say I build a sorting algorithm that is O(N^2) complexity and works fine for small inputs (takes <1 millisecond), but it is going to be used for large data systems. Suddenly it takes hundreds of thousands of hours to sort the data. One of the corps I worked with went full scalability in their architecture. One-click deployments, dynamic scaling of servers, rebalancing of databases, automatic provisioning of storage. They were handling 40-50k requests pr. second with their 15-ish large server farm, which could sale down to 5 servers, or up to 50-ish before it began to wobble. I got called in because the company had gotten a large client that needed 100k requests pr. second. They tried scaling the system to fit the need, but the whole thing got unstable and their solution was "more operations people to manage it". I built a custom solution for the backend. Took about two months. The new system could do about 2100k requests pr. second on one server. Scalability of the new system was ~90% efficient as well, so lots of capacity for the future. None of their developers understood computers or the science behind them. They were all educated and experienced developers, but none of that were applied to the problem. They were just assembling parts from the hardware store until something worked, and the resulting Frankenstein's Monster was put into production.
- wizofaus 4y ago
  I'm struggling to believe any single server could usefully service 2100k (well over 2 million!) requests per second. Even Google, with their vast farm of servers, reportedly only process less than 100k requests per second globally. I've certainly read of servers capable of handling in the order of 1000k requests per second as a benchmark, but the requests are usually pretty trivial (the one I saw literally did no input processing at all, and just returned a single fixed byte! But was written in Java, surprisingly.) At any rate, I would think a tiny % of real-life systems actually need to be able to support that sort of load, and bringing in somebody to do the scalability work once it's clear it's needed seems like exactly the right strategy to me.
  - Genbox 4y ago
    Not serving, but handling 2100k requests. Your skepticism is rightly placed, as the HTTP protocol is yet an example of an inefficient protocol that nonetheless is used as the primary protocol on the internet. Some webservers[1] can serve millions of requests pr. second, but I'd never use HTTP in code where efficiency is key. No, I'm talking about handling requests. In this particular case, requests (32 to 64 bytes) were flowing through several services (on the same computer). I replaced the processing chain with a single application to remove the overhead of serialization between processes. Requests were filtered early in the pipeline, which made a ~55% reduction in the work needed. Requests were then batched into succinct data structures and processed via SIMD. Output used to be JSON, but I instead wrote a custom memory allocator and just memcpy the entire blob on to the wire. Before: No pre-filtering, off-the-shelf databases (PSQL), queue system for I/O, named pipes and TCP/IP for local data transfer. Lots of concurrency issues, thread starvation and I/O bound work. After: Agnessive pre-filtering, succinct data structures for cache coherence, no serialization overhead, SIMD processing. Can saturate a 32 core CPU with almost no overhead. [1] https://www.techempower.com/benchmarks/#section=data-r13&hw=ph&test=plaintext https://www.techempower.com/benchmarks/#section=data-r13&hw=...