4 ms·
I think that this is an argument over apples and oranges. I'm well familiar with the write problem you're speaking of and even wrote a blog a couple of years ag
by gtrubetskoy 10y ago
I think that this is an argument over apples and oranges. I'm well familiar with the write problem you're speaking of and even wrote a blog a couple of years ago detailing influx's storage code.
The "apples" use case is when you want for lack of a better description a Grafana chart out of it. (E.g. the cluster cpu load, "we're getting X page views per second", etc).
The "oranges" case is when every data point matters, and I think that this particular case is not so much about time series, but logging, where a log is a "time series" technically speaking. Here of course you're talking massive amounts of data which you want to do your best to optimize and it's a hard problem. It's not a _time series_ problem, it's a problem of storage. Splunk (sorry, can't think of a better example ATM) doesn't describe itself as a "times series database", and yet it's addressing that very problem of horizontally scalable write-intensive storage. So is hbase, cassandra, accumulo, bigquery and all their friends.
In my experience the best solution is to use two separate systems one for each case. Because there is absolutely no value in a "disk used" measurement every millisecond - aggregated to once a minute (or few) is perfectly fine, and retaining the original data is not needed and you'd be a fool to build a Cassndra cluster to handle it. On the other hand if I'm recording equity bids and asks, well then I better record every point forever.
- bbrazil 10y agoThat's two separate dimensions. One is metrics vs. event logging. The other is consistency vs. availability. There's many types of logs, and you don't need the consistency that'd be required for billing-related logs as for debug logs. Similarly you may have billing-related metrics (e.g. bandwidth usage) where consistency is important. > It's not a _time series_ problem, it's a problem of storage. Technically anything with a time dimension is a time series, so Splunk is a time series database. > In my experience the best solution is to use two separate systems for each case. I agree completely. There's at least two general problems here that need different approaches once you get beyond trivial scale.
- siculars 10y agoRe. your horizontal scalable write intensive use case is basically what Riak TS was designed to handle, but specifically with a temporally sorted data model. Riak TS just got a version bump too [0]. Because Riak TS is a derivative of Riak KV it basically inherited the clustering, HA, bla, bla from day 0. /disclaimer/ I work for Basho. [0] http://docs.basho.com/riak/ts/1.5.0/releasenotes/ http://docs.basho.com/riak/ts/1.5.0/releasenotes/