4 ms·

I like applying the Circuit Breaker approach to caching. Rather than applying a cache everywhere, setup a cache wrapper that only triggers if the accessed resou

I like applying the Circuit Breaker approach to caching. Rather than applying a cache everywhere, setup a cache wrapper that only triggers if the accessed resource is slower than a certain threshold. The goal is for the cache to go unused most of the time, but during times of unexpected system stress the caches will kick in, provide relief to the system and reduce impact to users.

bawolff 4y ago
Having a different code path that only triggers during high load feels pretty risky to me.
- bell-cot 4y ago
  In theory, the best way to manage this situation may be with a grayzone - if all is well, then there's only a (say) 5% chance of going to cache. (To at least catch "the cache is broken" errors sooner.) As things get worse, that chance goes up as you cross-fade to mostly using the cache. Important Warning: If you do not have a very good understanding of your overall system, or neglect to carefully dampen the cross-fade functions - then the whole thing can demonstrate fast and unpredictable swings in behavior. Note that being subpoenaed to testify in a "Why did the Crater City chemical plant blow up?" investigation is usually bad for one's career.
  - hinkley 4y ago
    That's better but composition is always the devil in the details. Which might be what you mean when you say: > or neglect to carefully dampen the cross-fade functions People with a 'caching problem' don't have one cache, they have a bunch of them. So that 1 in 20 chance of getting grey zoned on one cache, is 1 in 400 chance of getting grey zoned on two caches, and a 1 in 8000 chance to get grey zoned on 3. That is a very small fraction but when you're dealing with the Law of Large Numbers that's something that's happening at least once a second for many services. But if you're using this you probably have a lot more than 3 caches, and then we have to talk about the odds that any request gets grey zoned at least once. which is going to be 1 - (19/20)^n. At six caches you're already over 1 in 4, which is going to potentially change your mean response time quite a bit.
    - bell-cot 4y ago
      > "...which is going to potentially change your mean response time quite a bit." My experience is that the changing mean response time isn't nearly as bad as the changing standard deviation. The really big problems tend to arise when you have unsuspected or under-appreciated positive feedback loops - whether in the software, or in associated physical systems and human behaviors. Or resonance effects. Those situations are where good dampening gets really important.
zasdffaa 4y ago
> The goal is for the cache to go unused most of the time Then you have a useful cache sitting there unused unless at peak time - why? Just enable it always. This makes no sense.
- brightball 4y ago
  Depends on how large the cache is and how much users expect instant updates to their data. If real-time or near real-time is the expectation, then avoiding the cache usage unless necessary preserves their experience without incurring the hurdles of cache invalidation that grow more complex the more you cache. EDIT: Another area where this approach is beneficial is with a multi-tenant system where larger tenants querying more data could have a much more significant performance impact than other tenants on the system. Time of day/week/month usage spikes can also cause a performance degrade for everyone even if it's not typically stressing the system. This approach is just a mitigation to prevent the system from getting overload during those times. It's not the right approach for every situation. Just a spin on the Circuit Breaker Pattern. https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern
  - salawat 4y ago
    I'm beginning to wonder whether my personal moniker of "signal propagation" is nominatively equivalent to what others call "cache invalidation". I.e. you cannot by definition have a state change until the signal thereof has traversed a link from A to B. No signal, no change. If you have a cache for one system that delays writes to a backing data store referenced by other systems that are unaware of that cache, then you have necessarily a partition problem. Every problem in a distributed computing context can generally be reduced to a signal propagation problem.
    - ilyt 4y ago
      It's only a problem if you're breaking deadlines. Like having someone's status in communication app update with 10s delay is probably not a a problem, so as long as you don't ever cache it for longer than that, it is entirely fine. > If you have a cache for one system that delays writes to a backing data store referenced by other systems that are unaware of that cache, then you have necessarily a partition problem. In most cases people mean caching reads so the problem here is stale reads, not delayed writes. It's not necessarily the same, some component caching (or essentially, queuing) write means that once the write is done, all readers (assuming they don't have any extra cache) see it at once, while with caching reads (and no invalidation), multiple readers can have different view
pyrolistical 4y ago
Sounds more like a dark buffer anti pattern
ilyt 4y ago
...instead of having better user experience all the time ? And have more code to manage on top of that ? Sorry but that sounds like terrible idea. And when system starts being slow the cache is empty, because you waited for system to slow down to start filling it
- brightball 4y ago
  If the system is fast enough without the cache this isn’t a problem. In order to overwhelm it, you need a cascade of stress at the same time. Caching as the stress appears prevents the issue. Databases already do plenty of in RAM caching with available free memory anyway, you just don’t get to manage it directly.

bawolff 4y ago

Having a different code path that only triggers during high load feels pretty risky to me.

bell-cot 4y ago
In theory, the best way to manage this situation may be with a grayzone - if all is well, then there's only a (say) 5% chance of going to cache. (To at least catch "the cache is broken" errors sooner.) As things get worse, that chance goes up as you cross-fade to mostly using the cache. Important Warning: If you do not have a very good understanding of your overall system, or neglect to carefully dampen the cross-fade functions - then the whole thing can demonstrate fast and unpredictable swings in behavior. Note that being subpoenaed to testify in a "Why did the Crater City chemical plant blow up?" investigation is usually bad for one's career.
- hinkley 4y ago
  That's better but composition is always the devil in the details. Which might be what you mean when you say: > or neglect to carefully dampen the cross-fade functions People with a 'caching problem' don't have one cache, they have a bunch of them. So that 1 in 20 chance of getting grey zoned on one cache, is 1 in 400 chance of getting grey zoned on two caches, and a 1 in 8000 chance to get grey zoned on 3. That is a very small fraction but when you're dealing with the Law of Large Numbers that's something that's happening at least once a second for many services. But if you're using this you probably have a lot more than 3 caches, and then we have to talk about the odds that any request gets grey zoned at least once. which is going to be 1 - (19/20)^n. At six caches you're already over 1 in 4, which is going to potentially change your mean response time quite a bit.
  - bell-cot 4y ago
    > "...which is going to potentially change your mean response time quite a bit." My experience is that the changing mean response time isn't nearly as bad as the changing standard deviation. The really big problems tend to arise when you have unsuspected or under-appreciated positive feedback loops - whether in the software, or in associated physical systems and human behaviors. Or resonance effects. Those situations are where good dampening gets really important.

zasdffaa 4y ago

> The goal is for the cache to go unused most of the time Then you have a useful cache sitting there unused unless at peak time - why? Just enable it always. This makes no sense.

brightball 4y ago
Depends on how large the cache is and how much users expect instant updates to their data. If real-time or near real-time is the expectation, then avoiding the cache usage unless necessary preserves their experience without incurring the hurdles of cache invalidation that grow more complex the more you cache. EDIT: Another area where this approach is beneficial is with a multi-tenant system where larger tenants querying more data could have a much more significant performance impact than other tenants on the system. Time of day/week/month usage spikes can also cause a performance degrade for everyone even if it's not typically stressing the system. This approach is just a mitigation to prevent the system from getting overload during those times. It's not the right approach for every situation. Just a spin on the Circuit Breaker Pattern. https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern
- salawat 4y ago
  I'm beginning to wonder whether my personal moniker of "signal propagation" is nominatively equivalent to what others call "cache invalidation". I.e. you cannot by definition have a state change until the signal thereof has traversed a link from A to B. No signal, no change. If you have a cache for one system that delays writes to a backing data store referenced by other systems that are unaware of that cache, then you have necessarily a partition problem. Every problem in a distributed computing context can generally be reduced to a signal propagation problem.
  - ilyt 4y ago
    It's only a problem if you're breaking deadlines. Like having someone's status in communication app update with 10s delay is probably not a a problem, so as long as you don't ever cache it for longer than that, it is entirely fine. > If you have a cache for one system that delays writes to a backing data store referenced by other systems that are unaware of that cache, then you have necessarily a partition problem. In most cases people mean caching reads so the problem here is stale reads, not delayed writes. It's not necessarily the same, some component caching (or essentially, queuing) write means that once the write is done, all readers (assuming they don't have any extra cache) see it at once, while with caching reads (and no invalidation), multiple readers can have different view

pyrolistical 4y ago

Sounds more like a dark buffer anti pattern

ilyt 4y ago

...instead of having better user experience all the time ? And have more code to manage on top of that ? Sorry but that sounds like terrible idea. And when system starts being slow the cache is empty, because you waited for system to slow down to start filling it

brightball 4y ago
If the system is fast enough without the cache this isn’t a problem. In order to overwhelm it, you need a cascade of stress at the same time. Caching as the stress appears prevents the issue. Databases already do plenty of in RAM caching with available free memory anyway, you just don’t get to manage it directly.