3 ms·

As someone who has worked on very large networks for 15 years, the author left out some important bits about using caching with thundering herd problems. You r

As someone who has worked on very large networks for 15 years, the author left out some important bits about using caching with thundering herd problems. You really need to make sure that your cache settings will not generate a ton of requests to the origin when new content is requested. If you have a bunch of clients that are going to request content at nearly the exact same time (this happens a lot with live video streaming, but can also happen when your clients are configured to download something as soon as it becomes available, or on a schedule, etc), you need to make sure your caches are set up so that the second request will wait for the first request to fill from origin rather than send its own cache fill request. Ideally, if 1000 clients all request the same content from your cache, only ONE request should go back to the origin, and the other 999 should wait and used the content retrieved by the first request. Otherwise, your cache isn't going to do anything since every request is going to require pulling from origin anyway. In nginx, you do this with something like the proxy_cache_lock setting.

bitexploder 3y ago
Caching is generally way harder to get right than a lot of people probably think. Having delved into building caching systems that get the behavior and needs of the system just right, it took way more effort and research than I thought it would. You start with something simple and by the time you have chased down all the edge cases and tuned the cache behavior to your situation it is often not so simple after all.
01HNNWZ0MV43FF 3y ago
I recall seeing that called "Request coalescing" in Faster Than Lime's article https://fasterthanli.me/articles/request-coalescing-in-async-rust#a-little-caching-can-t-hurt https://fasterthanli.me/articles/request-coalescing-in-async...
- oops 3y ago
  It is also sometimes called request collapsing.
  - pcthrowaway 3y ago
    It's also sometimes called request deduplication
omeze 3y ago
Yep, Ive found that, generally, caching logic should be a separate flow from business logic. E.g wrap your RPC logic such that an in-memory cache transparently sits in your RPC stack. This lets you implement locking on the cache in a single place, and makes it easy to do things like eager prefetching or offloading cache logic to another thread.

bitexploder 3y ago

Caching is generally way harder to get right than a lot of people probably think. Having delved into building caching systems that get the behavior and needs of the system just right, it took way more effort and research than I thought it would. You start with something simple and by the time you have chased down all the edge cases and tuned the cache behavior to your situation it is often not so simple after all.

01HNNWZ0MV43FF 3y ago

I recall seeing that called "Request coalescing" in Faster Than Lime's article https://fasterthanli.me/articles/request-coalescing-in-async-rust#a-little-caching-can-t-hurt https://fasterthanli.me/articles/request-coalescing-in-async...

oops 3y ago
It is also sometimes called request collapsing.
- pcthrowaway 3y ago
  It's also sometimes called request deduplication

omeze 3y ago

Yep, Ive found that, generally, caching logic should be a separate flow from business logic. E.g wrap your RPC logic such that an in-memory cache transparently sits in your RPC stack. This lets you implement locking on the cache in a single place, and makes it easy to do things like eager prefetching or offloading cache logic to another thread.