3 ms·

Yes, my experience (not much) is that what makes YouTube or Google or any of those products really impressive is the speed. YouTube or Google Search suggestion

by 101008 5y ago

Yes, my experience (not much) is that what makes YouTube or Google or any of those products really impressive is the speed. YouTube or Google Search suggestion is good, and I think it could be replicable with that amount of data. What is insane is the speed. I can't think how they do it. I am doing something similar for the company I work on and it takes seconds (and the amount of data isn't that much), so I can't wrap my head around it. The point is that doing only speed is not _that_ complicated, and doing some algorithms alone is not _that_ complicated. What is really hard is to do both.

ecnahc515 5y ago
A lot of this is just spending more money and resources to make it possible to optimize for speed. With sufficient caching with and a lot of parallelism makes this possible. That costs money though. Caching means storing data twice. Parallelism means more servers (since you'll probably be aiming to saturate the network bandwidth for each host). Pre-aggregating data is another part of the strategy, as that avoids using CPU cycles in the fast-path, but it means storing even more copies of the data! My personal anecdotal experience with this is with SQL on object storage. Query engines that use object storage can still perform well with the above techniques, even though querying large amounts of data from object is slow. You can bypass the slowness of object storage if you pre-cache the data somewhere else that's closer/faster for recent data. You can have materialized views/tables for rollups of data over longer periods of time, which reduces the data needed to be fetched and cached. It also requires less CPU due to working with a smaller amount of pre-calculated data. Apply this to every layer, every system, etc, and you can get good performance even with tons of data. It's why doing machine-learning in real- is way harder than pre-computing models. Streaming platforms make this all much easier as you can constantly be pre-computing as much as you can, and pre-filling caches, etc. Of course, having engineers work on 1% performance improvements in the OS kernel, or memory allocators, etc will add up and help a lot too.
- QuercusMax 5y ago
  One interesting thing to note is that there are lots of internal tools (CLI, web UI, etc.) that are REALLY slow. Things that are heavily used in the fast-path for development (e.g. code search, code review, test results) are generally pretty quick, but if there's a random system that has a UI, it's probably going to be very slow - because there's no budget for speeding them up, and the only people it annoys are engineers from other teams.
simcop2387 5y ago
I've had them take seconds for suggestions before when doing more esoteric searches. I think there's an inordinate amount of cached suggestions and they have an incredible way to look them up efficiently.
jiggawatts 5y ago
Latency. Latency. Latency! It's hard to measure, so nobody does. Throughput is easy to measure, so everybody does. Latency is hard to buy, so few people try. Throughput is easy to buy, so everybody does. Latency is what matters to every user. Throughput matters only to a few people. Turn on SR-IOV. Disable ACPI C-states. Stop tunnelling internal traffic through virtual firewalls. Use binary protocols instead of JSON over HTTPS. I've seen just those alone improve end-user experience tenfold.
- nomel 5y ago
  > Disable ACPI C-states. Related, I worked on an embedded linux device that bit-banged an unusual communication protocol. I wanted to offload this bit banging to some dedicated microcontrollers built into the SOC, to help free up the main CPU for some other use. I spent some time writing a nice little device driver that used DMA to send the transaction to the bit banger. The main CPU usage went from 60% down to 2%! Success! Unfortunately, the transactions/second was now a fraction of what it was before. After some digging, it turned out the improved CPU usage was low enough that the CPU scaling governor was staying at the lowest frequency. Disabling the governor and forcing the CPU to max frequency resulted in a pretty huge transactions/second improvements. Obvious in retrospect, of course.

by 101008 5y ago

ecnahc515 5y ago

A lot of this is just spending more money and resources to make it possible to optimize for speed. With sufficient caching with and a lot of parallelism makes this possible. That costs money though. Caching means storing data twice. Parallelism means more servers (since you'll probably be aiming to saturate the network bandwidth for each host). Pre-aggregating data is another part of the strategy, as that avoids using CPU cycles in the fast-path, but it means storing even more copies of the data! My personal anecdotal experience with this is with SQL on object storage. Query engines that use object storage can still perform well with the above techniques, even though querying large amounts of data from object is slow. You can bypass the slowness of object storage if you pre-cache the data somewhere else that's closer/faster for recent data. You can have materialized views/tables for rollups of data over longer periods of time, which reduces the data needed to be fetched and cached. It also requires less CPU due to working with a smaller amount of pre-calculated data. Apply this to every layer, every system, etc, and you can get good performance even with tons of data. It's why doing machine-learning in real- is way harder than pre-computing models. Streaming platforms make this all much easier as you can constantly be pre-computing as much as you can, and pre-filling caches, etc. Of course, having engineers work on 1% performance improvements in the OS kernel, or memory allocators, etc will add up and help a lot too.

QuercusMax 5y ago
One interesting thing to note is that there are lots of internal tools (CLI, web UI, etc.) that are REALLY slow. Things that are heavily used in the fast-path for development (e.g. code search, code review, test results) are generally pretty quick, but if there's a random system that has a UI, it's probably going to be very slow - because there's no budget for speeding them up, and the only people it annoys are engineers from other teams.

simcop2387 5y ago

I've had them take seconds for suggestions before when doing more esoteric searches. I think there's an inordinate amount of cached suggestions and they have an incredible way to look them up efficiently.

jiggawatts 5y ago

Latency. Latency. Latency! It's hard to measure, so nobody does. Throughput is easy to measure, so everybody does. Latency is hard to buy, so few people try. Throughput is easy to buy, so everybody does. Latency is what matters to every user. Throughput matters only to a few people. Turn on SR-IOV. Disable ACPI C-states. Stop tunnelling internal traffic through virtual firewalls. Use binary protocols instead of JSON over HTTPS. I've seen just those alone improve end-user experience tenfold.

nomel 5y ago
> Disable ACPI C-states. Related, I worked on an embedded linux device that bit-banged an unusual communication protocol. I wanted to offload this bit banging to some dedicated microcontrollers built into the SOC, to help free up the main CPU for some other use. I spent some time writing a nice little device driver that used DMA to send the transaction to the bit banger. The main CPU usage went from 60% down to 2%! Success! Unfortunately, the transactions/second was now a fraction of what it was before. After some digging, it turned out the improved CPU usage was low enough that the CPU scaling governor was staying at the lowest frequency. Disabling the governor and forcing the CPU to max frequency resulted in a pretty huge transactions/second improvements. Obvious in retrospect, of course.