4 ms·
Dear HN, recently time series DBs appear to have become very popular on HN but I'm confused. What are they for? I understand the idea for, say, physics experim
by skrebbel 10y ago
Dear HN, recently time series DBs appear to have become very popular on HN but I'm confused. What are they for?
I understand the idea for, say, physics experiments. Lots of parameters sampled thousands of times per second, gotta store that stuff somewhere. But this is HN, not an experimental physics subreddit, so I must be missing something.
What do people on HN who care about this stuff use time series databases for? Why are there 20+ competitors?
- leetbulb 10y agoDruid provides realtime ad-hoc campaign analytics to our advertisers and publishers for our performance CPA network.
- dsl 10y agoAs an example, I track requests per second per datacenter in a time series database. This allows me to make pretty graphs and do traffic engineering between sites.
- majke 10y agoWhen you run servers, you might want to record plenty of parameters. From received packets counters, via disk io operations, to cpu load. If you want to identify issues with good precision, you will record _hundreds_ of counters per each server. Sticking this kind of data to relational database is not good. With couple of servers you might have millions of _writes_ per second. While the data is only read when you refresh some charts. This makes the time series db unique - they need to handle write-heavy loads, and perform well when asked to retrieve couple of parameters within defined time window. So two things: - time series db's are needed for anyone running live systems and measuring them on-the-fly - time series db's have unique access patterns, not usually optimized by more conventional databases
- paukiatwee 10y agoTime serie DB for storing: Monitoring data, IoT data, etc. See example supported inputs by Telegraf https://github.com/influxdata/telegraf/tree/master/plugins/inputs https://github.com/influxdata/telegraf/tree/master/plugins/i... from InfluxData
- kejaed 10y agoThe world is generating massive amounts of data. Not all of it fits as easily into relational database schemes like a customer with and order that buys some widgets. Take, for example, fleet management data streaming in. Thousands of vehicles with time, GPS and other metrics streaming in. There's nothing terribly relational about this data, so there can be better-designed systems to handle ingress and analytics.
- threeseed 10y agoBig data analytics is the biggest new trend in enterprises right now. And one of the key parts of it is building machine learning models which can do things like look at a customer's previous behaviour in order to predict what their future behaviour might be. So this is one scenario where a time series database is infinitely better than say a RDBMS or Columnar database. Likewise performing analytics on IoT devices e.g. sensors in trucks or oil/gas equipment requires events to be captured, stored and later mined rapidly. And many time series database being schemaless allow you to manage data from disparate sources in the one table.
- thanatropism 10y ago> So this is one scenario where a time series database is infinitely better than say a RDBMS or Columnar database I still don't get it. Most of my background is in "small data" and database programs tend either to store time series in "objects" (other kinds of objects are models and graphs), like Eviews, or as ultimately-isomorphic-to-spreadsheets tables with some added syntactic sugar for time structure that's only understood by some functions. I'm beginning to do some "not so small data" (text analysis from news websites now; but the essential thing is the time sequencing of information cascades) with sqlite and pandas (pandas is just horrible, but it's already there) and basically the only problem I have is that raw data is unevenly sampled and I have to make some choices when downsampling to a fixed grid so statistical analysis proper can be performed. That said: I understand, as the grandparent poster, that in some cases (high-end physics experiments) time-structured data is incrementally produced in enormous quantities, and just storing a thousand variables at ten thousand samples per second is this whole challenge. But server logs? Sensor readings? At one point I had an Arduino hobby where I had a robot moving "of his own will" based on thermistors and did the necessary downsampling and filtering in the Arduino sketch itself for later analysis in Matlab. I mean, who's getting raw data from IoT devices into servers? Even my news scraping thing does some pre-selection before committing to the db. People underestimate how quickly the signal-to-noise ratio decays at high frequencies; and it seems to me that Cargo Cult IT is massively overestimating its need for CERN-level compute.
- gilbetron 10y ago
- michaelhoney 10y agoWe use InfluxDB to store solar energy performance for hundreds of locations at 15-minute intervals. It doesn't take long to get millions of points, and querying them efficiently is important.
- ddoran 10y agoTime series databases are critical in financial institutions, storing end of day (or much more frequent) marks for example. The advantage of such databases is not just their ability to store temporal data natively, but the built-in functions specific to time-based datasets which would not be atomic in a non-time-based db.
- perlgeek 10y agoGoogle basically does all its monitoring through time series databases. Basically the record response times, error rates, and internal health metrics from all kinds of services and service instances all as time series, and then define rules on top of these to define when to generate alerts. Other things you typically want to record are all kinds of resource usage metrics, like disk usage, RAM usage, number of processes, network traffic and saturation etc.
- a3n 10y agoInteresting to uncover data that's not directly related to a requirement: get some search results. Of course it's humans sitting at the laptop, so time matters. But the other data you mention also has the nature of "the thing we care about" along with "time associated with the thing we care about." I wonder what other extra-dimensional (totally made up term on the fly) data we might care about. Power per query. Temperature per query. Water consumed per query. Water consumed per current activity level of trending topics.
- perlgeek 10y agoMoney earned per... is usually a pretty big thing :-)
- parenthephobia 10y agoWe use it to store fine-grained server and application metrics for performance monitoring, bottleneck detection, and general troubleshooting.