4 ms·
It's a database for a metric platform. Think of OpenTSDB and Prometheus. Or for a better comparison think of Thanos https://thanos.io/ https://thanos.io/ As t
by buro9 7y ago
It's a database for a metric platform.
Think of OpenTSDB and Prometheus. Or for a better comparison think of Thanos https://thanos.io/ https://thanos.io/
As to whether they could fulfil Uber's needs, the thing about scale (real massive scale - I work at Cloudflare) is that everything breaks in weird ways according to your specific uses of a technology. The things listed above work for companies, until they don't. There's few things that seem to truly work at every scale, Kafka and ClickHouse come to mind for wholly different use cases than a time series database.
- 1996 7y ago> ClickHouse come to mind for wholly different use cases than a time series database. ClickHouse works fine as a TSDB if you don't mind getting a little dirty
- valyala 7y agoThere is a TSDB solution if you don't want getting a little dirty - VictoriaMetrics [1]. It is built on the same principles as ClickHouse [2]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast... [2] https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282 https://medium.com/@valyala/how-victoriametrics-makes-instan...
- 1996 7y ago"-retentionPeriod - retention period in months for the data. Older data is automatically deleted. Default period is 1 month." Not sure I want that in a TSDB! "Reducing disk space usage by deleting unneded time series. This doesn't work as expected, since the deleted time series occupy disk space until the next merge operation, which can never occur." Ouch. But ok, disk space is cheap. The killer point: it seems to be purely json based- no SQL of any kind. I'm not sure about that. A lot of code would have to be changed to fit that model.
- manigandham 7y agoClickhouse (and other columnstore RDBMS) are all perfectly fine for time-series and usually better than the standard options because they have SQL querying.
- linuxhansl 7y agoWe run OpenTSDB (which stores its data in Apache HBase) at scale. 150m-200m events/minute and about 20-30 trillion (10^12) events stored. Doubling about every 12-18 months or so. While it's true that things start to creak at scale, this has worked remarkably well for us so far. I doubt M3DB is somehow magical in this regard.
- roskilli 7y agoM3DB ingested 30 million datapoints per second (so 1.8 billion per minute) with each node writing hundreds of thousands of writes per second. The dataset was in the petabytes. For us the cost savings vs OpenTSDB (millions of dollars of hardware), the faster query time and the reduction in oncall overhead was worthwhile.
- shaklee3 7y agoHundreds of thousands per second isn't very high when you compare that to clickhouse or kdb+.