3 ms·
There is a significant storage overhead due to all of the data that is collected about the computation (the "dynamic dependency graph"). The storage overhead i
by assface 11y ago
There is a significant storage overhead due to all of the data that is collected about the computation (the "dynamic dependency graph").
The storage overhead is massive. It's a non-starter with this approach. Our experiments with self-adjusting computation were in the range of 30-100x for simple algorithms. That means if you have a 1TB data set, you need 30TB just to store the intermediate results.
A relational database with materialized views or the special purpose systems that you cite (DBToaster, LINVIEW) are better approaches.
- umutacar 11y agostorage overhead of self-adjusting computation depends on the granularity at which dependencies are tracked, which is in the control of the programmer. my post above provides more information.