3 ms·
That's pretty neat! Can these be used to efficiently compute rolling percentiles (over windows of the data), or just incremental?
by breuleux 5y ago
That's pretty neat! Can these be used to efficiently compute rolling percentiles (over windows of the data), or just incremental?
- deleted 5y ago[deleted]
- WireBaron 5y agoThe UDDSketch (default) implementation will allow rolling percentiles, though we still need a bit of work on our end to support it. There isn't a way to do this with TDigest however.
- jeffbee 5y agoSure there is. You simply maintain N phases of digests, and every T time you evict a phase and recompute the summary (because T-digests are easily merged).
- djk447 5y agoI think this would be a tumbling window rather than a true "rolling" tdigest. I suppose you could decrement the buckets, but it gets a little weird as splits can't really be unsplit. The tumbling window one would probably work, though Tdigest is a little weird on merge etc as it's not completely deterministic with respect to ordering and merging (Uddsketch is) so it's likely you get something that is more than good enough, but wouldn't be the same as if you just calculated it directly so it gets a little confusing and difficult. (NB: Post author here).
- cyral 5y agoThis is what I do, it's not a true rolling digest but it works well enough for my purposes.