4 ms·

I’m really curious how this works and how it’s implementation compares to something like materialize — I wonder if there are any caveats around consistency

nickvanw 4y ago
Great question! We have a technical blog post about how PlanetScale Boost is implemented: https://planetscale.com/blog/how-planetscale-boost-serves-your-sql-queries-instantly https://planetscale.com/blog/how-planetscale-boost-serves-yo... In short, it can be compared in consistency to an up-to-date read replica; PlanetScale Boost uses Vitess' VStream to process events as they happen and keep itself up to date. The blog has much more information if you're curious.
- deleted 4y ago
  [deleted]
- dang 4y ago
  (We've merged the threads so that blog post is now the URL at the top)
  - nickvanw 4y ago
    Thank you as always for everything that you do @dang!
  - deleted 4y ago
    [deleted]
- valbu 4y ago
  I think the SQL example should be bit more extreme, the count() group by is quite common and has just linear scaling and is plenty fast for majority of use cases. Tested with 1 process thread, 1k stars = 0.285ms; 10k = 2.85 ms; (and 0.1k = 45us that is same as overhead or just selecting 1 row without join and group by). So with 1k stars you need the system to average 3500 calls/s to saturate 1 thread or have meaningful latency impact. Sure, for bigger IO or row counts this does not scale and materialized view is indeed >100x faster.
giovannibonetti 4y ago
It seems similar to MIT's Noria [1] > Noria is a new streaming data-flow system designed to act as a fast storage backend for read-heavy web applications based on Jon Gjengset's Phd Thesis, as well as this paper from OSDI'18. It acts like a database, but precomputes and caches relational query results so that reads are blazingly fast. Noria automatically keeps cached results up-to-date as the underlying data, stored in persistent base tables, change. Noria uses partially-stateful data-flow to reduce memory overhead, and supports dynamic, runtime data-flow and query change. [1] https://github.com/mit-pdos/noria https://github.com/mit-pdos/noria

nickvanw 4y ago

Great question! We have a technical blog post about how PlanetScale Boost is implemented: https://planetscale.com/blog/how-planetscale-boost-serves-your-sql-queries-instantly https://planetscale.com/blog/how-planetscale-boost-serves-yo... In short, it can be compared in consistency to an up-to-date read replica; PlanetScale Boost uses Vitess' VStream to process events as they happen and keep itself up to date. The blog has much more information if you're curious.

deleted 4y ago
[deleted]
dang 4y ago
(We've merged the threads so that blog post is now the URL at the top)
- nickvanw 4y ago
  Thank you as always for everything that you do @dang!
- deleted 4y ago
  [deleted]
valbu 4y ago
I think the SQL example should be bit more extreme, the count() group by is quite common and has just linear scaling and is plenty fast for majority of use cases. Tested with 1 process thread, 1k stars = 0.285ms; 10k = 2.85 ms; (and 0.1k = 45us that is same as overhead or just selecting 1 row without join and group by). So with 1k stars you need the system to average 3500 calls/s to saturate 1 thread or have meaningful latency impact. Sure, for bigger IO or row counts this does not scale and materialized view is indeed >100x faster.

giovannibonetti 4y ago

It seems similar to MIT's Noria [1] > Noria is a new streaming data-flow system designed to act as a fast storage backend for read-heavy web applications based on Jon Gjengset's Phd Thesis, as well as this paper from OSDI'18. It acts like a database, but precomputes and caches relational query results so that reads are blazingly fast. Noria automatically keeps cached results up-to-date as the underlying data, stored in persistent base tables, change. Noria uses partially-stateful data-flow to reduce memory overhead, and supports dynamic, runtime data-flow and query change. [1] https://github.com/mit-pdos/noria https://github.com/mit-pdos/noria