4 ms·

That is one hell of a comprehensive article. I wonder how much impact would such extreme optimizations on a real-world application, which for example does DB qu

That is one hell of a comprehensive article. I wonder how much impact would such extreme optimizations on a real-world application, which for example does DB queries. This experiment feels similar to people who buy old cars and remove everything from the inside except the engine, which they tune up so that the car runs faster :).

101008 5y ago
Yes, my experience (not much) is that what makes YouTube or Google or any of those products really impressive is the speed. YouTube or Google Search suggestion is good, and I think it could be replicable with that amount of data. What is insane is the speed. I can't think how they do it. I am doing something similar for the company I work on and it takes seconds (and the amount of data isn't that much), so I can't wrap my head around it. The point is that doing only speed is not _that_ complicated, and doing some algorithms alone is not _that_ complicated. What is really hard is to do both.
- ecnahc515 5y ago
  A lot of this is just spending more money and resources to make it possible to optimize for speed. With sufficient caching with and a lot of parallelism makes this possible. That costs money though. Caching means storing data twice. Parallelism means more servers (since you'll probably be aiming to saturate the network bandwidth for each host). Pre-aggregating data is another part of the strategy, as that avoids using CPU cycles in the fast-path, but it means storing even more copies of the data! My personal anecdotal experience with this is with SQL on object storage. Query engines that use object storage can still perform well with the above techniques, even though querying large amounts of data from object is slow. You can bypass the slowness of object storage if you pre-cache the data somewhere else that's closer/faster for recent data. You can have materialized views/tables for rollups of data over longer periods of time, which reduces the data needed to be fetched and cached. It also requires less CPU due to working with a smaller amount of pre-calculated data. Apply this to every layer, every system, etc, and you can get good performance even with tons of data. It's why doing machine-learning in real- is way harder than pre-computing models. Streaming platforms make this all much easier as you can constantly be pre-computing as much as you can, and pre-filling caches, etc. Of course, having engineers work on 1% performance improvements in the OS kernel, or memory allocators, etc will add up and help a lot too.
  - QuercusMax 5y ago
    One interesting thing to note is that there are lots of internal tools (CLI, web UI, etc.) that are REALLY slow. Things that are heavily used in the fast-path for development (e.g. code search, code review, test results) are generally pretty quick, but if there's a random system that has a UI, it's probably going to be very slow - because there's no budget for speeding them up, and the only people it annoys are engineers from other teams.
- simcop2387 5y ago
  I've had them take seconds for suggestions before when doing more esoteric searches. I think there's an inordinate amount of cached suggestions and they have an incredible way to look them up efficiently.
- jiggawatts 5y ago
  Latency. Latency. Latency! It's hard to measure, so nobody does. Throughput is easy to measure, so everybody does. Latency is hard to buy, so few people try. Throughput is easy to buy, so everybody does. Latency is what matters to every user. Throughput matters only to a few people. Turn on SR-IOV. Disable ACPI C-states. Stop tunnelling internal traffic through virtual firewalls. Use binary protocols instead of JSON over HTTPS. I've seen just those alone improve end-user experience tenfold.
  - nomel 5y ago
    > Disable ACPI C-states. Related, I worked on an embedded linux device that bit-banged an unusual communication protocol. I wanted to offload this bit banging to some dedicated microcontrollers built into the SOC, to help free up the main CPU for some other use. I spent some time writing a nice little device driver that used DMA to send the transaction to the bit banger. The main CPU usage went from 60% down to 2%! Success! Unfortunately, the transactions/second was now a fraction of what it was before. After some digging, it turned out the improved CPU usage was low enough that the CPU scaling governor was staying at the lowest frequency. Disabling the governor and forcing the CPU to max frequency resulted in a pretty huge transactions/second improvements. Obvious in retrospect, of course.
talawahtech 5y ago
This comprehensive level of extreme tuning is not going to be directly useful to most people; but there are a few things in there like SO_ATTACH_REUSEPORT_CBPF that I hope to see more servers and frameworks adopt. Similarly I think it is good to be aware of the adaptive interrupt capabilities of AWS instances, and the impacts of speculative execution mitigations, even if you stick to the defaults. More importantly it is about the idea of using tools like Flamegraphs (or other profiling tools) to identify and eliminate your bottlenecks. It is also just fun to experiment and share the results (and the CloudFormation template). Plus it establishes a high water mark for what is possible, which also makes it useful for future experiments. At some point I would like to do a modified version of this that includes DB queries.
- BiteCode_dev 5y ago
  What does SO_ATTACH_REUSEPORT_CBPF and how does one uses it?
  - bboreham 5y ago
    That is covered in the article.
- longhairedhippy 5y ago
  Wow, I haven't seen SO_ATTACH_REUSEPORT_CBPF before, I didn't even know it existed. That is a pretty ingenious and powerful primitive to cut down on cross-NUMA chatter. I always like it when folks push things to the extreme, it really shows exactly what is going on under the hood.
mkoubaa 5y ago
Speaking of which I wonder if anyone did this to the Linux kernel for a variant that's tuned only for http
- astrange 5y ago
  He's cheating by assuming all http responses fit in one TCP packet, but you could use FreeBSD which is already tuned like this and has optimizations like ACCEPT_FILTER_HTTP not mentioned in this article.

101008 5y ago

Yes, my experience (not much) is that what makes YouTube or Google or any of those products really impressive is the speed. YouTube or Google Search suggestion is good, and I think it could be replicable with that amount of data. What is insane is the speed. I can't think how they do it. I am doing something similar for the company I work on and it takes seconds (and the amount of data isn't that much), so I can't wrap my head around it. The point is that doing only speed is not _that_ complicated, and doing some algorithms alone is not _that_ complicated. What is really hard is to do both.

ecnahc515 5y ago
A lot of this is just spending more money and resources to make it possible to optimize for speed. With sufficient caching with and a lot of parallelism makes this possible. That costs money though. Caching means storing data twice. Parallelism means more servers (since you'll probably be aiming to saturate the network bandwidth for each host). Pre-aggregating data is another part of the strategy, as that avoids using CPU cycles in the fast-path, but it means storing even more copies of the data! My personal anecdotal experience with this is with SQL on object storage. Query engines that use object storage can still perform well with the above techniques, even though querying large amounts of data from object is slow. You can bypass the slowness of object storage if you pre-cache the data somewhere else that's closer/faster for recent data. You can have materialized views/tables for rollups of data over longer periods of time, which reduces the data needed to be fetched and cached. It also requires less CPU due to working with a smaller amount of pre-calculated data. Apply this to every layer, every system, etc, and you can get good performance even with tons of data. It's why doing machine-learning in real- is way harder than pre-computing models. Streaming platforms make this all much easier as you can constantly be pre-computing as much as you can, and pre-filling caches, etc. Of course, having engineers work on 1% performance improvements in the OS kernel, or memory allocators, etc will add up and help a lot too.
- QuercusMax 5y ago
  One interesting thing to note is that there are lots of internal tools (CLI, web UI, etc.) that are REALLY slow. Things that are heavily used in the fast-path for development (e.g. code search, code review, test results) are generally pretty quick, but if there's a random system that has a UI, it's probably going to be very slow - because there's no budget for speeding them up, and the only people it annoys are engineers from other teams.
simcop2387 5y ago
I've had them take seconds for suggestions before when doing more esoteric searches. I think there's an inordinate amount of cached suggestions and they have an incredible way to look them up efficiently.
jiggawatts 5y ago
Latency. Latency. Latency! It's hard to measure, so nobody does. Throughput is easy to measure, so everybody does. Latency is hard to buy, so few people try. Throughput is easy to buy, so everybody does. Latency is what matters to every user. Throughput matters only to a few people. Turn on SR-IOV. Disable ACPI C-states. Stop tunnelling internal traffic through virtual firewalls. Use binary protocols instead of JSON over HTTPS. I've seen just those alone improve end-user experience tenfold.
- nomel 5y ago
  > Disable ACPI C-states. Related, I worked on an embedded linux device that bit-banged an unusual communication protocol. I wanted to offload this bit banging to some dedicated microcontrollers built into the SOC, to help free up the main CPU for some other use. I spent some time writing a nice little device driver that used DMA to send the transaction to the bit banger. The main CPU usage went from 60% down to 2%! Success! Unfortunately, the transactions/second was now a fraction of what it was before. After some digging, it turned out the improved CPU usage was low enough that the CPU scaling governor was staying at the lowest frequency. Disabling the governor and forcing the CPU to max frequency resulted in a pretty huge transactions/second improvements. Obvious in retrospect, of course.

talawahtech 5y ago

This comprehensive level of extreme tuning is not going to be directly useful to most people; but there are a few things in there like SO_ATTACH_REUSEPORT_CBPF that I hope to see more servers and frameworks adopt. Similarly I think it is good to be aware of the adaptive interrupt capabilities of AWS instances, and the impacts of speculative execution mitigations, even if you stick to the defaults. More importantly it is about the idea of using tools like Flamegraphs (or other profiling tools) to identify and eliminate your bottlenecks. It is also just fun to experiment and share the results (and the CloudFormation template). Plus it establishes a high water mark for what is possible, which also makes it useful for future experiments. At some point I would like to do a modified version of this that includes DB queries.

BiteCode_dev 5y ago
What does SO_ATTACH_REUSEPORT_CBPF and how does one uses it?
- bboreham 5y ago
  That is covered in the article.
longhairedhippy 5y ago
Wow, I haven't seen SO_ATTACH_REUSEPORT_CBPF before, I didn't even know it existed. That is a pretty ingenious and powerful primitive to cut down on cross-NUMA chatter. I always like it when folks push things to the extreme, it really shows exactly what is going on under the hood.

mkoubaa 5y ago

Speaking of which I wonder if anyone did this to the Linux kernel for a variant that's tuned only for http

astrange 5y ago
He's cheating by assuming all http responses fit in one TCP packet, but you could use FreeBSD which is already tuned like this and has optimizations like ACCEPT_FILTER_HTTP not mentioned in this article.