3 ms·
Your intuition is correct. Incremental computation is fundamentally a time-space tradeoff. Depending on the views you write, you might end up maintaining large
by lsuresh 2y ago
Your intuition is correct. Incremental computation is fundamentally a time-space tradeoff. Depending on the views you write, you might end up maintaining large amounts of state. We've written about it here: https://www.feldera.com/blog/streaming-needs-storage https://www.feldera.com/blog/streaming-needs-storage
That said, Feldera has several features to keep state bounded even when computing on infinite streams. For example, we do automatic garbage collection (GC) where with some static analysis, we can figure out when it is safe to forget inputs that will no longer affect the output of views.
We recently ported a community member's warehouse workload to Feldera where with these features, we were evaluating As-Of joins and streaming aggregations with 1.2GB of RAM on a laptop with more than a million events/sec in perf.