5 ms·

> DO NOT say things like "we should add a cache to reduce latency” We should add a cache to reduce latency tho…

jeron 4y ago
yeah, not sure what is too hard to understand about that. Perhaps it depends on the context but I highly doubt you can go wrong by saying a cache should be added to reduce latency.
- N_A_T_E 4y ago
  Only two hard things in computer science, blah blah.
- lhorie 4y ago
  The way it typically fall aparts for my candidates is I ask about optimizations, they say X could be cached, then I ask about cache invalidation and lo and behold, it didn't even occur to them that the cache needs to be invalidated at some point, so they blurt something along the lines of doing the entire expensive operation again and checking it against the cache, or something similarly nonsensical. More common than I'd like to admit.
  - ThalesX 4y ago
    > then I ask about cache invalidation No doubt with a follow up on how the variable would be best named.
    - old_hat 4y ago
      That's obvious. It's called "x".
      - Apocryphon 4y ago
        You mean x1
  - nonethewiser 4y ago
    Isn’t it better to know you can add a cache to reduce latency, even if you’re not sure how to invalidate it?
    - lhorie 4y ago
      The problem isn't demonstrating understanding of what a cache does conceptually, in fact any SWE worth their salt ought to be able to explain caching. The problem is if the candidate's decision to add a cache causes the system behavior to become obviously incorrect upon an ounce of further inspection, because this demonstrates a lack of foresight about local maxima vs global maxima, and systems are all about trade-offs at the macro scale. Caching is an easy thing to blurt out in an interview setting, but not all problem spaces benefit from caching and caching often isn't the only available solution, or the most ideal one.
    - Misdicorl 4y ago
      I started typing no, but then.... I honestly can't decide which would be worse: a developer who literally doesn't know what a cache is or one who installs caches with bad invalidation policies.
- mgraczyk 4y ago
  In practice, "latency" is too vague to be what matters. In many systems it's the worst 1% or 0.1% of latencies that matter. Adding a cache will improve latency in those cases only in some situations. Usually adding caches helps by reducing load on the service behind the cache.
gilbetron 4y ago
If you need to reduce average latency and the cache would actually do that, sure. Cache doesn't magically make all your network latencies drop, though.
mgraczyk 4y ago
What I am saying here is that it's not true in all cases, so candidates should avoid blurting out rehearsed sentences like that. Caches usually reduce load on the thing behind the cache. They can sometimes reduce latency in ways that matter, but often don't. For example if the p99 requests all miss in cache, then the cache won't help.
- Kranar 4y ago
  But that's not true, the total latency including a cache is: P(cache_hit) * cache_read_latency + (1 - P(cache_hit)) * cache_miss_latency Assuming your scenario of 99% cache misses, that means you need a cache whose read latency is less than 1% of the cost of a cache miss. There are plenty of ways to design such a cache so that you still get a net performance benefit, even in your greatly exaggerated scenario. One very simple example of an incredibly cheap cache that is almost certainly going to cost less than 1% of the cache miss latency is a bloom filter. Bloom filters can be tuned to be incredibly space efficient as well. https://en.wikipedia.org/wiki/Bloom_filter https://en.wikipedia.org/wiki/Bloom_filter Please don't be so dismissive of people who answer questions by giving generally good and standard advice, just because you have some silly corner case that you want to play "gotcha" on.
  - mgraczyk 4y ago
    You are talking about minimizing the expected or average latency, which is very rarely important. The actual goal will depend on the broader system but typically you want to minimize the latency of the worst 95%, 99%, 99.9% etc of requests. Whether or not the cache reduces latency depends on the distribution of the cache hits. In practice, items that miss a simple LRU cache often have the longest latency in other parts of the system, in which case the cache would not reduce p99 latency. EDIT: I recommend reading the Google SRE book or the famous "tale at scale" article. https://www.barroso.org/publications/TheTailAtScale.pdf https://www.barroso.org/publications/TheTailAtScale.pdf
    - Kranar 4y ago
      Well folks, you heard it here. Average latency is rarely important and if someone presents a very standard and common way to reduce latency of a system, you should dismiss them because they should be aware that your specific system that they have no knowledge of would result in 99% of cache misses. I used to work at Google long ago in the platforms division, in fact I worked on the BigTable cache (among other things, mostly related to performance). It would be very sad indeed if today's SRE book dismisses caching as a vital and standard optimization strategy and instead plays all kinds of gotchas with potential candidates as I believe you're doing. With that said, you article you linked does not in any way support your claim about caching, on the contrary it hardly discusses caching at all. It's as if you just wanted to dump a document you thought I wouldn't read as a way to be dismissive.

yeah, not sure what is too hard to understand about that. Perhaps it depends on the context but I highly doubt you can go wrong by saying a cache should be added to reduce latency.

N_A_T_E 4y ago
Only two hard things in computer science, blah blah.
lhorie 4y ago
The way it typically fall aparts for my candidates is I ask about optimizations, they say X could be cached, then I ask about cache invalidation and lo and behold, it didn't even occur to them that the cache needs to be invalidated at some point, so they blurt something along the lines of doing the entire expensive operation again and checking it against the cache, or something similarly nonsensical. More common than I'd like to admit.
- ThalesX 4y ago
  > then I ask about cache invalidation No doubt with a follow up on how the variable would be best named.
  - old_hat 4y ago
    That's obvious. It's called "x".
    - Apocryphon 4y ago
      You mean x1
- nonethewiser 4y ago
  Isn’t it better to know you can add a cache to reduce latency, even if you’re not sure how to invalidate it?
  - lhorie 4y ago
    The problem isn't demonstrating understanding of what a cache does conceptually, in fact any SWE worth their salt ought to be able to explain caching. The problem is if the candidate's decision to add a cache causes the system behavior to become obviously incorrect upon an ounce of further inspection, because this demonstrates a lack of foresight about local maxima vs global maxima, and systems are all about trade-offs at the macro scale. Caching is an easy thing to blurt out in an interview setting, but not all problem spaces benefit from caching and caching often isn't the only available solution, or the most ideal one.
  - Misdicorl 4y ago
    I started typing no, but then.... I honestly can't decide which would be worse: a developer who literally doesn't know what a cache is or one who installs caches with bad invalidation policies.
mgraczyk 4y ago
In practice, "latency" is too vague to be what matters. In many systems it's the worst 1% or 0.1% of latencies that matter. Adding a cache will improve latency in those cases only in some situations. Usually adding caches helps by reducing load on the service behind the cache.

gilbetron 4y ago

If you need to reduce average latency and the cache would actually do that, sure. Cache doesn't magically make all your network latencies drop, though.

mgraczyk 4y ago

What I am saying here is that it's not true in all cases, so candidates should avoid blurting out rehearsed sentences like that. Caches usually reduce load on the thing behind the cache. They can sometimes reduce latency in ways that matter, but often don't. For example if the p99 requests all miss in cache, then the cache won't help.

Kranar 4y ago
But that's not true, the total latency including a cache is: P(cache_hit) * cache_read_latency + (1 - P(cache_hit)) * cache_miss_latency Assuming your scenario of 99% cache misses, that means you need a cache whose read latency is less than 1% of the cost of a cache miss. There are plenty of ways to design such a cache so that you still get a net performance benefit, even in your greatly exaggerated scenario. One very simple example of an incredibly cheap cache that is almost certainly going to cost less than 1% of the cache miss latency is a bloom filter. Bloom filters can be tuned to be incredibly space efficient as well. https://en.wikipedia.org/wiki/Bloom_filter https://en.wikipedia.org/wiki/Bloom_filter Please don't be so dismissive of people who answer questions by giving generally good and standard advice, just because you have some silly corner case that you want to play "gotcha" on.
- mgraczyk 4y ago
  You are talking about minimizing the expected or average latency, which is very rarely important. The actual goal will depend on the broader system but typically you want to minimize the latency of the worst 95%, 99%, 99.9% etc of requests. Whether or not the cache reduces latency depends on the distribution of the cache hits. In practice, items that miss a simple LRU cache often have the longest latency in other parts of the system, in which case the cache would not reduce p99 latency. EDIT: I recommend reading the Google SRE book or the famous "tale at scale" article. https://www.barroso.org/publications/TheTailAtScale.pdf https://www.barroso.org/publications/TheTailAtScale.pdf
  - Kranar 4y ago
    Well folks, you heard it here. Average latency is rarely important and if someone presents a very standard and common way to reduce latency of a system, you should dismiss them because they should be aware that your specific system that they have no knowledge of would result in 99% of cache misses. I used to work at Google long ago in the platforms division, in fact I worked on the BigTable cache (among other things, mostly related to performance). It would be very sad indeed if today's SRE book dismisses caching as a vital and standard optimization strategy and instead plays all kinds of gotchas with potential candidates as I believe you're doing. With that said, you article you linked does not in any way support your claim about caching, on the contrary it hardly discusses caching at all. It's as if you just wanted to dump a document you thought I wouldn't read as a way to be dismissive.