3 ms·

> At a high level, Polaris interacts with a stateful service as a client and assumes no knowledge of the service internals. This allows it to be generic. We hav

> At a high level, Polaris interacts with a stateful service as a client and assumes no knowledge of the service internals. This allows it to be generic. We have dozens of Polaris integrations at Meta. “Cache should eventually be consistent with the database” is a typical client-observable invariant that Polaris monitors, especially in the presence of asynchronous cache invalidation. In this case, Polaris pretends to be a cache server and receives cache invalidation events. I'm a bit confused here. Polaris behaves as a regular cache server which receives invalidation events, but isn't the typical bug related to cache invalidation that a service forgets to contact the cache server for invalidation? So this will only catch cases where (1) you remember to contact Polaris, but you forgot to contact other cache servers [which Polaris happens to know about], OR (2) you're not handling errors during invalidation requests to the cache server [and the request to Polaris was successful]? Or are you cache servers "smart" and might have internal logic which "ignores" an invalidation? What am I missing? EDIT: Reading through "A real bug we found and fixed this year" and I'm still a bit confused. It seems like a very contrived bug directed directly to how you deal with versioning (e.g. you allow the latest version to be present with stale metadata, or something?). My main concern with cache invalidation is what to invalidate at what time.

uvdn7 4y ago
> isn't the typical bug related to cache invalidation that a service forgets to contact the cache server for invalidation? That's not the case based on our experience at FB. At-least-once delivery is a solved problem basically. But you are absolutely right that if there's an issue in the invalidation delivery, it's possible that polaris won't receive the event as well. Polaris actually supports a separate event stream (all the way from client initiated writes) to cover this case.
- benlivengood 4y ago
  It helps that TAO is a write through cache so clients can't really forget to invalidate, correct? If someone were to directly write to MySQL shards there would be stale data ~indefinitely. I'm assuming the versioning is what ensures proper write, invalidation, and fetch ordering so that e.g. slow mysql writes/replication don't cause remote TAO clusters to receive an invalidation message and then read a stale value from a replica?
  - uvdn7 4y ago
    > It helps that TAO is a write through cache so clients can't really forget to invalidate, correct? That's not the case. > If someone were to directly write to MySQL shards there would be stale data For the sake of this discussion, you can assume this is the setup, and everything should still stand. > I'm assuming the versioning is what ensures proper write, invalidation, and fetch ordering so that e.g. slow mysql writes/replication don't cause remote TAO clusters to receive an invalidation message and then read a stale value from a replica? You should join us :p This is getting into the details of our cache invalidation protocol and our cache consistency model. Maybe another post on another day!
uvdn7 4y ago
> EDIT: Reading through "A real bug we found and fixed this year" and I'm still a bit confused. yeah that specific bug is convoluted. > It seems like a very contrived bug directed directly to how you deal with versioning (e.g. you allow the latest version to be present with stale metadata, or something?). This is mostly for performance reason that I won't go into details too much here. Version is mostly an internal concept that clients don't care about. So it's OK-ish. > My main concern with cache invalidation is what to invalidate at what time. Can you elaborate? Our experience on cache invalidation is that you invalidate on mutation, and in terms of "what to invalidate" it depends on the computation, which you might have to give me a little more details.
elliotswart 4y ago
I did an analysis of the bug from the blog post and the TAO paper, and from what I can tell the fundamental bug is therefore the fact that the error handler that handles cache invalidation errors drops only lesser versions, while the cache invalidation contract requires replacing equal or lesser versions. There is a detailed write up here: https://elliotswart.github.io/pragmaticformalmodeling/caching/reproducing-the-bug/ https://elliotswart.github.io/pragmaticformalmodeling/cachin...

uvdn7 4y ago

> isn't the typical bug related to cache invalidation that a service forgets to contact the cache server for invalidation? That's not the case based on our experience at FB. At-least-once delivery is a solved problem basically. But you are absolutely right that if there's an issue in the invalidation delivery, it's possible that polaris won't receive the event as well. Polaris actually supports a separate event stream (all the way from client initiated writes) to cover this case.

benlivengood 4y ago
It helps that TAO is a write through cache so clients can't really forget to invalidate, correct? If someone were to directly write to MySQL shards there would be stale data ~indefinitely. I'm assuming the versioning is what ensures proper write, invalidation, and fetch ordering so that e.g. slow mysql writes/replication don't cause remote TAO clusters to receive an invalidation message and then read a stale value from a replica?
- uvdn7 4y ago
  > It helps that TAO is a write through cache so clients can't really forget to invalidate, correct? That's not the case. > If someone were to directly write to MySQL shards there would be stale data For the sake of this discussion, you can assume this is the setup, and everything should still stand. > I'm assuming the versioning is what ensures proper write, invalidation, and fetch ordering so that e.g. slow mysql writes/replication don't cause remote TAO clusters to receive an invalidation message and then read a stale value from a replica? You should join us :p This is getting into the details of our cache invalidation protocol and our cache consistency model. Maybe another post on another day!

uvdn7 4y ago

> EDIT: Reading through "A real bug we found and fixed this year" and I'm still a bit confused. yeah that specific bug is convoluted. > It seems like a very contrived bug directed directly to how you deal with versioning (e.g. you allow the latest version to be present with stale metadata, or something?). This is mostly for performance reason that I won't go into details too much here. Version is mostly an internal concept that clients don't care about. So it's OK-ish. > My main concern with cache invalidation is what to invalidate at what time. Can you elaborate? Our experience on cache invalidation is that you invalidate on mutation, and in terms of "what to invalidate" it depends on the computation, which you might have to give me a little more details.

elliotswart 4y ago

I did an analysis of the bug from the blog post and the TAO paper, and from what I can tell the fundamental bug is therefore the fact that the error handler that handles cache invalidation errors drops only lesser versions, while the cache invalidation contract requires replacing equal or lesser versions. There is a detailed write up here: https://elliotswart.github.io/pragmaticformalmodeling/caching/reproducing-the-bug/ https://elliotswart.github.io/pragmaticformalmodeling/cachin...