3 ms·
In a non-streaming db What would prevent you from storing the result set and just using the last iteration to calculate the next?
by hobs 4y ago
In a non-streaming db What would prevent you from storing the result set and just using the last iteration to calculate the next?
- Lemaxoxo 4y agoThis is definitely a possibility. What I meant to say is that the implementation in the blog post doesn't support that.
- hobs 4y agoMakes sense, just an insert instead of a select, I used a similar approach on newton-raphson to implement XIRR in SQL and it worked well.
- swasheck 4y agoi’m trying to learn here so please pardon my ignorance. wouldn’t a pre-aggregated set affect the new aggregate result? i suppose you could store avg, sum, count and then add the new value(s) to sum, new count to count, and recalculate average from that. or even just avg and count and then re-aggregate as ((avg*count)+new values)/(count + new values count) but i didn’t know if there’s a better way to process new values into a set of data that’s already been aggregated
- hobs 4y agoYep that would be what I would do - effectively a functional approach where no memory is required besides the current set and iteration. A big part of materializing datasets for performance is finding the right grain that is both easy enough to calculate and also can do nice things like be resumeable for snapshots.