4 ms·

Someone knowledgeable might know: is this just incremental view updates? To what extent is the cache intelligent if parameters, where clauses, or aggregates cha

by thom 3y ago

Someone knowledgeable might know: is this just incremental view updates? To what extent is the cache intelligent if parameters, where clauses, or aggregates change? I really love this space and have been impressed with Materialize, but even if you can make some intermediate state incremental, if your workload is largely dynamic you end up needing to jump the whole way to OLAP platforms. I’m hopeful that we’re closer and closer to having our cake and eating it here, and that the operational data warehouse is only round the corner.

hobofan 3y ago
They have a bit about their technical foundation here[0]. Given that Readyset was co-founded by Jon Gjengset (but has apparently since departed the company), who authored the paper on Noria[1], I would assume that Readyset is the continuation of that research. I wouldn't call that "just" incremental view maintenance, as it's a lot more involved than the simplest implementation of IVM (though obviously that is the end-goal). So it shares some roots with Materialize. They have a common conceptual ancestry in Naiad, where Materialize evolved out of timely-dataflow. [0]: https://docs.readyset.io/concepts/streaming-dataflow https://docs.readyset.io/concepts/streaming-dataflow [1]: https://jon.thesquareplanet.com/papers/osdi18-noria.pdf https://jon.thesquareplanet.com/papers/osdi18-noria.pdf [2]: https://dl.acm.org/doi/10.1145/2517349.2522738 https://dl.acm.org/doi/10.1145/2517349.2522738 [3]: https://github.com/TimelyDataflow/timely-dataflow https://github.com/TimelyDataflow/timely-dataflow
- Jonhoo 3y ago
  Just poking my head in to say that I technically never departed ReadySet — what happened was that I co-founded the company, but was so burnt-out when it came to databases after my PhD that I decided to leave the running of the company to others. Then, US visa regulations made it so that I couldn't really be involved _at all_ if I wasn't an actual employee, which meant I truly was "just a founder" with no real involvement in the company's execution if you will. Now that I'm back in Europe, that's changing a bit, and I have regular calls with the CEO and such!
  - AYBABTME 3y ago
    Thanks so much for your work! Visa problems are a bane on the US. I hope that this situation will improve for you and get better in the future.
- jauntywundrkind 3y ago
  Wow, uh, Noria got listed a bunch of times looking back in search. https://hn.algolia.com/?q=noria https://hn.algolia.com/?q=noria (It turns into Nokia results part way through page 2, but still a lot of submissions!)
lukoktonos 3y ago
Readyset auto-parameterizes cached queries similar to a prepared statement. If you run the same query with different parameters, it will be routed to the cache. The first time a parameter set is queried, it will be a cache miss and trigger an "upquery" to populate the cache, after which that set of parameters will be served from the cache. Different where clauses (sets of parameters) would map to different query-caches and currently need to be cached separately. Aggregates supported by Readyset[1] will also be automatically maintained, but depending on the query, they may be handled by post-processing after retrieving results from a cache [1] https://docs.readyset.io/reference/features/queries#aggregations https://docs.readyset.io/reference/features/queries#aggregat...

by thom 3y ago

hobofan 3y ago

They have a bit about their technical foundation here[0]. Given that Readyset was co-founded by Jon Gjengset (but has apparently since departed the company), who authored the paper on Noria[1], I would assume that Readyset is the continuation of that research. I wouldn't call that "just" incremental view maintenance, as it's a lot more involved than the simplest implementation of IVM (though obviously that is the end-goal). So it shares some roots with Materialize. They have a common conceptual ancestry in Naiad, where Materialize evolved out of timely-dataflow. [0]: https://docs.readyset.io/concepts/streaming-dataflow https://docs.readyset.io/concepts/streaming-dataflow [1]: https://jon.thesquareplanet.com/papers/osdi18-noria.pdf https://jon.thesquareplanet.com/papers/osdi18-noria.pdf [2]: https://dl.acm.org/doi/10.1145/2517349.2522738 https://dl.acm.org/doi/10.1145/2517349.2522738 [3]: https://github.com/TimelyDataflow/timely-dataflow https://github.com/TimelyDataflow/timely-dataflow

Jonhoo 3y ago
Just poking my head in to say that I technically never departed ReadySet — what happened was that I co-founded the company, but was so burnt-out when it came to databases after my PhD that I decided to leave the running of the company to others. Then, US visa regulations made it so that I couldn't really be involved _at all_ if I wasn't an actual employee, which meant I truly was "just a founder" with no real involvement in the company's execution if you will. Now that I'm back in Europe, that's changing a bit, and I have regular calls with the CEO and such!
- AYBABTME 3y ago
  Thanks so much for your work! Visa problems are a bane on the US. I hope that this situation will improve for you and get better in the future.
jauntywundrkind 3y ago
Wow, uh, Noria got listed a bunch of times looking back in search. https://hn.algolia.com/?q=noria https://hn.algolia.com/?q=noria (It turns into Nokia results part way through page 2, but still a lot of submissions!)

lukoktonos 3y ago

Readyset auto-parameterizes cached queries similar to a prepared statement. If you run the same query with different parameters, it will be routed to the cache. The first time a parameter set is queried, it will be a cache miss and trigger an "upquery" to populate the cache, after which that set of parameters will be served from the cache. Different where clauses (sets of parameters) would map to different query-caches and currently need to be cached separately. Aggregates supported by Readyset[1] will also be automatically maintained, but depending on the query, they may be handled by post-processing after retrieving results from a cache [1] https://docs.readyset.io/reference/features/queries#aggregations https://docs.readyset.io/reference/features/queries#aggregat...