14 ms·
Why Not to Build a Time-Series Database
- statictype 8y agoNice article. >its not uncommon for some of our customers to send us millions of metrics every minute What kind of customers/services generate millions of points a minute?
- capkutay 8y agoCustomers using a large-scale analytics database...
- jamescostian 8y agoSome companies record as much as they can, e.g. "here's all of the user's mouse-movements, their full User Agent, etc." or "here's everywhere the user tapped on the app and which of the notifications we sent them that they responded to, and more!" The example from the article was "one team at one of our customers decided to dump 30 million metrics on us, send all of their mobile product metrics into Outlyer"
- perfmode 8y ago1 mil is only 16K per second.
- andrewbinstock 8y agoTransportation companies--truck fleet data, airplanes, etc.
- JJMcJ 8y agoAt one employer most customers would opt-in to upload their system logs, and company would analyze to anticipate problems for preventive maintenance. The result was a tens of terabytes a day Niagara of data. When I left they were in early stages of Hadoop because ordinary parse/analyze was starting to fall behind.
- varikin 8y agoI use to work at a Fortune 50 retailer on the cloud platform (a lot of tooling around CI/CD for the teams that manage the website). We had a large problem with keeping the metrics pipeline current. A major issue is that be default, Spring Boot publishes about 500 different metrics on a 10 second slice. Allowing every application to pump out that many default metrics, most of which are never used, means that it takes only 334 instances to get one million metrics per minute (1,000,000 / 6 / 500 = 333 1/3). I would guess we had a couple thousand instances in production on a normal day. In a couple weeks, they have Black Friday. Any team that hasn't been able to fix performance problems are given the go ahead to just throw money at it and scale horizontally in obscene ways. Of course, our metrics were all handled in house. From talking to the teams that handled the metrics pipeline, the vendors were great for smaller companies, but there was no off the shelf solution for a companies that large with that volume. But I did very little with that myself, other than look into the fact that Spring Boot published way too many default metrics. Who needs P50, P70, P75, P80, P85, P90 - P99 on all web requests?! Just set a default that is small and worthwhile and let the developers adjust as needed.
- djrobstep 8y agoFolks need to resist the inclination to just gather maximum data for the hell of it. If you're pumping out a million metrics per minute, almost none of those are ever going to actually be used to generate meaningful insight.
- LukeShu 8y agoI used to work at a startup that made physical robots. The robot generated several GBs of data every time it turned on. You're correct, most of that data wasn't looked at most of the time. But every now and then, someone would say "Hey, I saw a robot do something funny the other day, what the hell happened?" And having all that data usually made it possible to figure out what happened. To me, "maximum data for the hell of it" isn't about generating insight by looking at trends, it's about generating insight during incident analysis.
- 8y ago
- azernik 8y agoI used to work at Meraki. There, we would, every 5 minutes or so, record byte and packet counts classified by traffic class and remote end of the connection for every single client connected to an SMB internal or customer-facing network. (Meraki actually did implement its own time-series database, and after I left published a paper describing its design and implementation. https://meraki.cisco.com/lib/pdf/trust/lt-paper.pdf https://meraki.cisco.com/lib/pdf/trust/lt-paper.pdf. Good quote on the motivation: "As discussed in Section 2.3.3, customers have a nearly insatiable demand for high-resolution historical data, even though they mostly query data from the recent past."
- jandrewrogers 8y agoMachine-generated event telemetry from mobile phones, cars, etc can easily be tens of millions per second in real-world applications. Human-generated event telemetry (e.g. text messaging) peaks at hundreds of thousands per second if you are working on global scales. There is virtually an unlimited number of applications that could generate 16k events per second (million per minute).
- kwillets 8y agoOften per-click stuff ends up with dozens or hundreds of data points from different parts of the code -- heartbeats, feature usage, funnels, experiment entry, etc.
- postwait 8y agoWe have customers that generate tens of millions of measurements per second. Lots of low-level systems latencies can be collected at high volume. Also, high volume online services can easily generate this order of magnitude.
- bbrazil 8y ago16k samples/s is not a lot. There are many Prometheus users with hundreds of thousands of samples/s on a single Prometheus server. Across their organisations it can be much more, Fastly has reported 2.2M/s (https://promcon.io/2018-munich/slides/monitoring-at-scale-migrating-to-prometheus-at-fastly.pdf https://promcon.io/2018-munich/slides/monitoring-at-scale-mi...) for example.
- OldHand2018 8y agoIt's pretty ridiculous that "Time-Series Database" has come to mean ingesting massive amounts of streaming data. They've been around a long time and have many use cases. They're a great way to store data efficiently, accessing specific data if you know the time range you are looking for is very fast and simple, and you can roll your own in a few dozen lines of C if that's what you want to do. If that's all you need, why not?
- davidjc1 8y agoThat may be a perfectly good solution if you have a very static infrastructure and narrow use case. As a thought exercise, for the most trivial solution, you could create a single append only flat file. This may work well for writes, but what happens when you want to read the datapoints for only a single series in time order? This would result in an expensive scan over the whole file. An improvement could be to create a file per series, but this becomes problematic when writing many small datapoints across each different file. The problem worsens in the case of a dynamic containerised infrastructure which produce a unique number of timeseries over very short intervals, which was the catalyst for the development of Prometheus TSDB v2, as the prior version stored a file per timeseries. As the post states, there is a balance between the read and write pattern - achieving that with a few lines of C for a general purpose case is a difficult task, if not impossible.
- OldHand2018 8y agoTo be clear, my post was to state that there are many use cases for time-series databases and bemoan the fact that most current development centers around a specific use case. That is in fact what I wrote. I have a hard time believing that "a dynamic containerised infrastructure which produce a unique number of timeseries over very short intervals" is the superset of all time-series use cases, but perhaps it is so.
- manigandham 8y ago"time-series database" is some of the most overhyped nonsense since noSQL. Time-series is just data with time as a primary component. It comes in all shapes and volume, but if you have a lot of data and are running heavy OLAP queries than we already have an entire class of capable databases. Use any modern distributed relational column-oriented database, set primary key to metric id + timestamp, and you'll be able to scale easily with full SQL and joins. You can keep your other business data there too, along with JSON, geospatial, window functions, and all the other rich analytical queries available with relational databases. We have trillion row tables that work great. No special "TSDB" needed.
- ghc 8y agoWhile I understand your point, you are quite mistaken if you think that time is just another key. Dealing with time properly requires a concept of point distance, similar to GIS systems requiring 2d distance understanding. You cannot do joins on time with SQL databases unless you want to throw away important data. As an example, in the industry I work in, you may have no readings for days or weeks, and then hundreds of readings from the same sensor. Why? Many systems in industrial environments send new readings only "on-change", and assume the underlying data storage architecture will forward fill to in-between times. This is why the practically ancient time series architecture of data historians still dominates in these use cases. In fact, for many time series functions you actually have to throw away the notion of relational joins to be able to efficiently perform time-preserving joins. Window functions only work in basic use cases with relatively small amounts of data where you're aggregating.
- deleted 8y ago[deleted]
- manigandham 8y agoI'm not sure where the complexity is that you claim, nor what it has to do with data historians? So what if there are missing rows? This doesn't affect the database and any aggregations will work fine. Databases don't "fill-in" data, but you can definitely write whatever SQL you need to create averages and buckets to smooth out results. From reading your website, it seems you're talking about the "last value recorded" as of a certain time, which doesn't seem to be a common query but is totally possible. KDB+ has "asof" joins and others can handle it with window functions using last_value(). We run queries on a table containing 2.7+ trillion rows of data that has no set pattern and infinite cardinality, and results return within seconds. Window functions and joins work without issue. Have you actually tried using a columnstore?
- camel_gopher 8y agoIn case anyone is interested in a video on this, here's the talk presented at the local San Francisco monitoring group, monitorSF (https://www.meetup.com/MonitorSF/ https://www.meetup.com/MonitorSF/) https://youtu.be/lA85vs6e3UA https://youtu.be/lA85vs6e3UA
- marshf 8y agoTime-series data handling/storage seems a mostly solved problem in the mining, oil, and manufacturing industries. Deployed in the field since the 80's, https://www.osisoft.com/about-osisoft/#more-about-pi-system https://www.osisoft.com/about-osisoft/#more-about-pi-system Discourse: Industry user now OSIsoft employee
- bigger_cheese 8y agoYes I can confirm data historians like PI are used heavily in my industry (manufacturing).
- cheriot 8y agoA solved problem at what scale? A kid with pencil and paper can solve the problem when it's small enough.
- seanhunter 8y agoI don't have personal knowledge, but my understanding is that timeseries data in the oil industry is insane scale. Think 10s or 100s of wells in a field, sensors at regular distances in every well each taking multiple readings (temperature, flow etc) on a sub-second cadence. And a big oil company will of course have many fields. It's in the same league as timeseries data in financial markets, which I have more direct experience of. Financial timeseries data probably has more metadata (trade/order flags etc) but in terms of raw data it's similar.
- cheriot 8y agoWhat the article is talking about is an order of magnitude larger. Think millions of data points per minute coming out of a medium sized data center. It gets harder when you look at the cardinality of the data (explained better by the article), which corresponds to the sized of the indexes.
- bra-ket 8y agoWe used a combination of Kafka + Hbase+ Phoenix (http://phoenix.apache.org/ http://phoenix.apache.org/) for similar purpose. It takes some effort to setup initial Hbase cluster but once you do it manually once and automate with Ansible /systemd it's pretty robust in operation. All our development was around query engine using plain JDBC/SQL to talk to Hbase via Phoenix. Scaling is as simple as adding a node in the cluster.
- PopeDotNinja 8y agoDoes Kafka have timestamps? I didn't see any when I looked, but I was working with an older client version & didn't get far into it.
- bra-ket 8y agowe didnt rely on it, or on the ordering of messages received from kafka, timestamps and transaction IDs were generated by the client app/kafka publisher and were part of the message put into kafka topic, when we consume that message with one of the parallel kafka consumers and save a row in hbase table that original timestamp + transactionId becomes part of the rowkey string, other parts being attributes that we wanted to index (secondary indices are supported in hbase/phoenix but we didnt use them too much, basically the composite rowkey is the index). Then when querying hbase it works as a parallel scanning machine and can do a time range scan + filtering +aggregation very fast. On a separate note we didn't use joins even though they are supported in Phoenix, data was completely denormalized into one big table.
- pritambarhate 8y agoThat's interesting. What are query times like? Let's say for single series to query data for a week at a five-minute interval, how many seconds it would take?
- objektif 8y agoNot very knowledgable in the area but can someone please explain how does kdb fit within this class of time series dbs and whether there are any alternatives available to kdb.
- tuukkah 8y agoTLDR: "Why Not to Build a Time-Series Database? Because we're building one and you should pay us." > Hopefully our story will make you think twice before trying to build your own TSDB in house using open-source solutions, or if you’re really crazy, building a TSDB from scratch. Building and maintaining a TSDB is a full time job, and we have dedicated expert engineers who are constantly improving and maintaing our TSDB, and no doubt will iterate the architecture again over time as we hit an even higher magnitude of scale down the line. > Given our experience in this complex space, I would sincerely recommend you don’t try and do this at home, and if you have the money you should definitely outsource this to the experts who do this as a full time job, whether its Outlyer or another managed TSDB solution out there. As so many things turn out in computing, it’s harder than it looks!
- peatfreak 8y ago"Don't do it because it's hard! And you should listen to us because we have the meta-knowledge and experience (now) to know everything there is to know about this topic, plus the bravery to admit in public that we are only human and we make mistakes." Isn't this the mantra of all of these types of articles? OK, yes, it usually makes sense. Especially in the case where you are like these guys and experienced enough in some relevant area to know just how difficult it can be. These are perfectly good reasons from technical, business, and project planning perspectives. Isn't there a TLDR where they mentioned when it does make sense to build your own TSBD? Presumably in some case where you have a team of serious, high-grade experts who know exactly what they are doing; have requirements that cannot be met by any of the other offerings out there; and where the whole thing has been specced out and deemed reasonable?
- User23 8y agoThat's how I read it too. To people who haven't worked with metrics at scale though there is some good information and it's worth reading. It blows my mind that businesses are willing to outsource metrics. When I worked at Amazon it was trivial to estimate the next quarter's results from the app metrics. Naturally this meant we were subject to trading restrictions. If a monitoring company ever starts applying Google/Facebook style ethics with regards to exploiting the data their customers give them, they will be in an incredibly powerful position.
- manish_gill 8y agoAs I was reading through the post I kept wondering why they weren't using some warehousing technique for older data - either dump it to S3 or better yet, Google BigQuery, which is amazingly fast at that scale. They only did it after doing lots of fire-fighting and per-tenant clusters. Clickhouse would also be a good option for doing aggregating queries that TSDBs are mostly used for. One of my wishlist items in the data space is a Managed Clickhouse offering. :-)
- dgildeh 8y agoAs the blog author, great to see the discussion and feedback, so appreciate it! Without going through comments one by one, the main ones about this being a solved problem or there's already solutions out there that do this, I would just say those comments remind me exactly of the type of conversations I had years ago with my team. We all thought it would be much easier or thought there would be something off the shelf that could do everything, and after several years of fire fighting, the reality was the problem looks much simpler than it really is, by a long mile. Now that we've been doing this for a few years, and spoken directly with creators of many other TSDBs, we take a very skeptical view of all claims made about any database. They all sound amazing when you first read about them, maybe even work great in testing, till you hit scale and then you find all the limitations. If there was a perfect TSDB out there, everyone would be using it and there wouldn't be a new one announced on a weekly basis! I think the one comment on query loads being different sums things up - I've no doubt all the other options thrown out there work well for data historians, but for monitoring tools with loads of concurrent users, loading dashboards with 10's or 100's of queries each, and alerting systems polling every few seconds in parallel, the query load can get very high quickly, and making those fast while still writing metrics in at scale, is a hard problem and I don't think any individual TSDB has really solved that properly, which is why we ended up building our own distributed architecture ourselves.
- Daneel_ 8y agoI’m surprised no one has brought up Splunk in here (that I could see at a cursory glance). They manage to do time-series storage on a pretty large scale (over 5PB/day for their largest customer).
- viraptor 8y agoI wouldn't really say that splunk is time-series in the same way they wanted. It's mainly logs indexing, but not the kind of aggregation you'd want from numeric, labelled metrics.