3 ms·

I also was surprising to not see the solution to the cache invalidation. The blog post describes how the tracing/logging mechanism works, but what about the act

I also was surprising to not see the solution to the cache invalidation. The blog post describes how the tracing/logging mechanism works, but what about the actual cache consistency/invalidation strategy? I.e. what about many-to-many where one fetch returns inconsistent results.. how do you even know it's inconsistent? What about data queried while there's a transaction rollback.. when is it updated in the cache? Sorry if it sounds like silly questions, I was genuinely curious to learn about the strategy since the blog started with "When it comes to cache invalidation, we believe we now have an effective solution to bridge the gap between theory and practice"

uvdn7 4y ago
These are good questions. I might not have satisfying answers to all of them but I will try. > what about many-to-many where one fetch returns inconsistent results Are you referring to the case when cache stores a complicated result of a function that fetches data from many sources, and how to figure out when to invalidate? If so, I am sorry but that's not the cache invalidation I was referring to. (https://news.ycombinator.com/item?id=31676102 https://news.ycombinator.com/item?id=31676102) > What about data queried while there's a transaction rollback.. when is it updated in the cache This might be easier to answer. I assume if you run the database with isolation level higher than read-committed, you will be fine.
- i_r7al 4y ago
  > delaying performing the compute-intensive operation until an inconsistent sample crosses the reporting timescale Could this delay cause a wrong reporting? Since the source of truth (the DB) could already received multiple mutate events of the data Polaris trying to verify? How do you handle this case?
  - uvdn7 4y ago
    No actually. When a sample has crossed the reporting timescale, and it's inconsistent. Regardless of if the db changes later or cache becomes consistent later, at the moment it's still an anomaly (inconsistency in this case).
uvdn7 4y ago
https://blog.the-pans.com/when-and-how-to-invalidate-cache/ https://blog.the-pans.com/when-and-how-to-invalidate-cache/ Does this include the "actual cache consistency/invalidation strategy" you were referring to?

uvdn7 4y ago

These are good questions. I might not have satisfying answers to all of them but I will try. > what about many-to-many where one fetch returns inconsistent results Are you referring to the case when cache stores a complicated result of a function that fetches data from many sources, and how to figure out when to invalidate? If so, I am sorry but that's not the cache invalidation I was referring to. (https://news.ycombinator.com/item?id=31676102 https://news.ycombinator.com/item?id=31676102) > What about data queried while there's a transaction rollback.. when is it updated in the cache This might be easier to answer. I assume if you run the database with isolation level higher than read-committed, you will be fine.

i_r7al 4y ago
> delaying performing the compute-intensive operation until an inconsistent sample crosses the reporting timescale Could this delay cause a wrong reporting? Since the source of truth (the DB) could already received multiple mutate events of the data Polaris trying to verify? How do you handle this case?
- uvdn7 4y ago
  No actually. When a sample has crossed the reporting timescale, and it's inconsistent. Regardless of if the db changes later or cache becomes consistent later, at the moment it's still an anomaly (inconsistency in this case).

uvdn7 4y ago

https://blog.the-pans.com/when-and-how-to-invalidate-cache/ https://blog.the-pans.com/when-and-how-to-invalidate-cache/ Does this include the "actual cache consistency/invalidation strategy" you were referring to?