7 ms·
MetricsDB: TimeSeries Database for storing metrics at Twitter
- DSingularity 6y ago1.2 PB saved using gorilla compression! Didn’t realize how sparse this data was.
- cbsmith 6y agoNot necessarily sparse. Just lacking in entropy. You can compress an arithmetic series down to a handful of bytes...
- hodgesrm 6y agoCompression for correlated measurements can be spectacular. ClickHouse can often get 99.9% reduction in data size with the right combination of codecs (delta, double-delta, gorilla, etc.) and compression (e.g., ZSTD). Sort order within tables is also important.
- akulkarni 6y agoCompression isn't just about sparse-ness, but optimizations one can make based on the dataset. E.g., Gorilla uses delta-delta-encoding for time-stamps, and a novel XOR-based compression scheme for floats. For more, here is a primer on Gorilla and other leading time-series algorithms: https://blog.timescale.com/blog/time-series-compression-algorithms-explained/ https://blog.timescale.com/blog/time-series-compression-algo...
- valyala 6y agoBTW, there are algorithms with better compression rate for time series data than Gorilla - see https://medium.com/faun/victoriametrics-achieving-better-compression-for-time-series-data-than-gorilla-317bc1f95932 https://medium.com/faun/victoriametrics-achieving-better-com...
- etaioinshrdlu 6y agoThis applies to all time series databases... What is special about time series that makes unique databases attractive? Is it the assumption that only 1 main index is created (time)? Is is that the data is append-only? In particular, I'd love to know if theres anything major that generic RDBMS's could do better here.
- dima_vm 6y agoYes, time-dependence and only numbers allows to use a lot of tricks that cannot be done with generic data. For example, double delta compression, or Gorilla for floating point numbers. For more, take a look at our open source VictoriaMetrics database, which uses all of such tricks.
- sharpy 6y agoYep. The nature of the time series data does allow for storage optimizations like that. But there is the usage pattern aspect as well, that is more interesting in my opinion. - Availability preferred over consistency (you want your metrics when bad things are happening, even if they may not be 100% accurate) - Extremely write heavy, with most data never being read (they mentioned that only 2% of data is ever read. in my experience at another large company, it was way less than 2%) - For the data that is read, most of reads are for most recent points (mostly alarms, but some dashboards as well) - Different SLAs for queries based on usage - ie. queries for alarms must be fast. Dashboards, and trend analysis - not so much.
- citrin_ru 6y agoClickHouse DBMS allows to combine delta, double delta, Gorilla with LZ4 or ZSTD for column compression. But it is not often used as DB for monitoring metrics so something else is probably expected from time series DB.
- redis_mlc 6y agoIt would work, but Clickhouse is a Russian (Yandex) thing, and SQL isn't really needed for most monitoring and alerting use cases. If I didn't want to use MySQL or Postgres, I'd rank Prometheus #1 and Clickhouse #2. The killer thing for Clickhouse is that Percona supports it, so if you want to outsource the installation, mgmt. and support, you can just write a check and get good results. Also, Clickhouse is a column store with SQL, so you could use an instance for monitoring and another to replace Vertica or Greenplum or whatever so long as it has the client libraries you need.
- polskibus 6y agoIs this open source? How does it differ from TimescaleDB or InfluxDB?
- redis_mlc 6y agoMetricsDB appears to be HDFS-based. TimescaleDB is Postgres-based and InfluxDB is memory and log-based. Prometheus is file-based.
- sytse 6y agoThanks for this clear summary of the differences.
- squarecog 6y agoWhile HDFS is indeed used for exporting old data and storing some partition mapping metadata, it's clear from the blog post that MetricsDB is much more reliant on BlobStore as well as MetricsDB-specific services. > The servers checkpoint in-memory data every two hours to durable storage, Blobstore. We are using Blobstore as durable storage so that our process can be run on our shared compute platform with lower management overhead.
- risyachka 6y agoAs fas as I have seen, 99% of in-house solutions are tailored for exactly 1 use case - the company that built it. Existing solutions like InfluxDB are made way more generic. Companies like Twitter, Uber etc can spend many millions and won't notice so they can build tailored databases for themselves. Will someone else use it? Probably not (sure unless this is something new or that differs a lot from existing solutions and gives a lot of benefits)
- privateprofile 6y agoI think it has (sadly) become common for people/companies to adopt software just/mostly because it came from Twitter/FAANG or whatever company that is high on hype...
- 6y ago
- nitred 6y agoCould someone tell me the disadvantages of solving the problem in the article using S3? I am actually curious.
- redis_mlc 6y agoThey're inserting 5 billion events per second (for some reason), and AWS throttles everything (the following link mentions a limit of 3,500/second): https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/ https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3... Also, if they're planning to do alerting, S3 would not help with that. PUTs are 1 cent per 2000: https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/
- buremba 6y ago5 billion events per second? That seems to be too much for monitoring data even at Twitter's scale. The article refers that they store the raw data and use rollup tables for aggregate queries, I wonder how much the ELT process costs in total.
- redis_mlc 6y ago> 5 billion events per second? When you work for a large company, you have 2 problems: 1) Every has-been manager bikesheds over metrics retention 2) you can't say no. See #1.
- dan-robertson 6y agoIndeed, the article mentions 5b metrics per minute
- deepakhj 6y agoYou write to S3 in chunks for long term storage and keep hot data live. You don’t write every metric directly. Along with compression It shouldn’t be that 5 billion/s.
- 6y ago
- wolf550e 6y agoA blog post by Dan Luu ('luu) about using this to save twitter a lot of money on RAM for JVMs: https://danluu.com/metrics-analytics/ https://danluu.com/metrics-analytics/
- manigandham 6y agoAnd yet another company that builds its own (time-series) database. Why does this keep happening for such a narrowly defined use-case at similar scales? At 1.5 petabytes, they could just dump this in Google's BigQuery for the cost of an engineer and have full SQL power.
- rco8786 6y agoBigquery does not have anything close to real time query support like you want/need for observability data. And plus the cost is more like one engineer per query. That shit is crazy expensive.
- manigandham 6y agoWhat do you mean by real-time? BQ can scan a petabyte in less than a minute, but partitioning and clustering will get most simple queries back in a few seconds. If you mean data freshness than BQ supports streaming ingest and will show those rows in the next query. SQL also lets you do a lot of aggregations in the database instead of pulling back raw metrics. And BQ has flat-rate pricing and discounts for large clients.
- buremba 6y agoTry to run a few petabyte-scale queries and you will be surprised when you see the invoice. If you switch to flat-rate pricing, don't expect the latency that you mentioned.
- manigandham 6y agoYou dont need to scan the whole dataset. Partitioning + clustering, especially for time-series, should be very efficient. BQ flat-rate pricing starts at 10k/month for a fixed number of processing slots. You can buy more slots to fit your query load. Anyways the point is that it's well within the financial means of Twitter, especially when compared to developing and operating this proprietary system.
- kasey_junk 6y agoStill waiting for someone to open source an observability db that uses distributions as the core storage abstraction instead of just single values. I thought Circonus’ IronDB was open source but they’ve either brought it back closed source or I was mistaken. In any case it seems weird that everyone is just rehashing the same space with metrics dbs and not moving to where Google/Circonus have been for a while.
- ekabod 6y agoclickhouse as aggregate functions (Pearson correlation coefficient,quantiles , etc... )[0]. Associated with their Materialized view and Liveview features, you can achieve observability [1]. [0]https://clickhouse-docs.readthedocs.io/en/latest https://clickhouse-docs.readthedocs.io/en/latest [1]https://www.altinity.com/blog/2019/11/13/making-data-come-to-life-with-clickhouse-live-view-tables https://www.altinity.com/blog/2019/11/13/making-data-come-to... Edit: fixed broken link.
- alexhutcheson 6y agoVeneur[1] can compute approximate global percentiles (among other things) during metric collection, and the percentiles can be stored and queried in even in a datastore that doesn't know anything about distributions. [1] https://github.com/stripe/veneur https://github.com/stripe/veneur
- kasey_junk 6y ago
- abeppu 6y agoI'd be curious to see a couple of organizations that have taken different approaches in this area discuss how they made decisions around investing in observability systems. When metrics gets to be a meaningful fraction of your total data size, and when it gets to be an operational burden on its own, I think it's tempting to ask "Do we really need to emit and store all this?" especially if you acknowledge that only ~2% of writes ever get read. In some sense, the organization is making an expected value of information estimate, where there's a complex interaction between the data you have and actions you can take, especially to preempt or resolve issues.
- hodgesrm 6y agoI've seen two approaches. (We have a lot of customers who deal with this exact problem.) 1.) Down-sample data to aggregates. Keep the aggregated but drop the source data after a certain period of time. You can see historical trends down to the sampling interval but cannot drill down to individual observations. 2.) Use tiered storage. Keep recent data on NVMe SSD and migrate to high density HDD or object storage over time. This allows you to keep more data. Not all DBMS support this out of the box.
- ssv33 6y agoCan someone offer up how this is different than InfluxDB? Also a bunch of machine log folks are jumping on the observability 'marketing' bandwagon - i.e. Elastic Search, Splunk, Devo, SumoLogic etc. Interestingly don't see them here..
- deppp 6y agoIf there is interest, we're working on time-series data labeling tool. Sneak peek at it here: https://htx-pub.s3.amazonaws.com/Screen+Shot+2020-06-17+at+17.56.23.png https://htx-pub.s3.amazonaws.com/Screen+Shot+2020-06-17+at+1... If you think it might be useful and you want to try it out drop me an email michael @ heartex.net