8 ms·
The Rise of Open Source Time Series Databases
- baggiponte 2y agoInteresting! Too bad it’s just about two of them.
- gmuslera 2y agoA bit opinionated and partial version of history (graphite was big before Influxdb and Prometheus came out, and the landscape is bigger than what they mentioned) but it might be a good enough starting point to learn about this.
- axytol 2y agoI'm using VictoriaMetrics (VM) to store basic weather data, like temperature and humidity. My initial setup was based on Prometheus however it seemed very hard to set a high data retention value, default was something like 15 days if I recall correctly. Since I would actually like to store all recorded values permanently, I could partially achieve this with VM which let me set a higher threshold, like 100 years. Still not 'forever' as I would have liked, but I guess me and my flimsy weather data setup will have other things than the retention threshold to worry about in 100 years. Would be nice to learn the reason why an infinite threshold is not allowed.
- starkparker 2y agoUsually performance and storage concerns. You can set effectively infinite retention on Prometheus, but after a long enough period you're going to like querying it even less. Most TSDBs aren't built for use cases like "query hourly/daily data over many years". Many use cases aren't looking further than 30 days because they're focused on things like app or device uptime and performance metrics, and if they are running on longer time frames they're recording (or keeping, or consolidating data to) far fewer data points to keep performance usable.
- atombender 2y agoMimir [1] is what we use where I work. We are very happy with it, and we have very long retention. Previously, our Prometheus setup was extremely slow if you went past today, but Mimir partitions the data to make it extremely fast to query even long time periods. We also used Thanos for a while, but Mimir apparently worked better. [1] https://grafana.com/oss/mimir/ https://grafana.com/oss/mimir/
- PeterZaitsev 2y agoYeah. Would be interested to see how VictoriaMetrics compares to Mimir, not just Prometheus. To be fair many projects in Prometheus "long term store" space come and gone - Thanos, Cortex, M3
- Too 2y agoCan you elaborate a bit more on the come and gone part?
- mtndew4brkfst 2y agoThey're out of tech posters' zeitgeist but AFAIK they are each still maintained and fulfilling people's needs. Just not as much commentary or front-of-mind-share.
- saswatamcode 2y agoHi! None of these are "gone" in any way, and are actually used in various different enterprises at global scale.
- mgxxx 2y agoThanos is well alive and kicking, actually used in some of the biggest infrastructure setups in the world such as Shopify and Cloudflare.
- hagen1778 2y agoHere you go https://victoriametrics.com/blog/mimir-benchmark/ https://victoriametrics.com/blog/mimir-benchmark/ It is from Sep 2022, it would be great to get newer results.
- starkparker 2y agoThis is unsurprisingly a VictoriaMetrics post that frames the history in a narrow way to talk favorably about VictoriaMetrics.
- to11mtm 2y agoCould be worse. I remember when a bunch of WSO2 contribs broke off, made their own (AGPL) ESB, and created a 'ESB Performance' website that cherry picked specific scenarios to make WSO2 look worse than their product. This was made funnier because the 'consultant' (really a buddy of management that used us as an experiment for all sorts of things, at least the code was readable, learned lots of languages fixing things lol) wound up having us go with the AGPL product, and it wasn't until I said to the team 'hey, lets try using the WSO2 docs as a cross reference' before we got productive. I do miss the 'dramatic readings' of UltraESB docs, learned a colleague understood Pashto [0]. [0] - After a specific 'verse', the colleague pointed out one of the odd phrases in the docs that prompted this practice, was a colloquialism in that language.
- Too 2y agoConveniently forgetting to mention any of the other big Open Source players in the field. Cortex, Graphite, Mimir, Thanos, M3, TimescaleDB.
- thebeardisred 2y agoThe author calls out InfluxDB as the first time series database. Why would that not have been rrdtool or something earlier?
- deleted 2y ago[deleted]
- gillh 2y agoThis topic has been done to death. Too many OSS options in the last ~10 years with very little differentiation. Let's talk LLMs instead.
- jnordwick 2y agoIf you are writing an article about time series databases, and you don't mention KDB - straight to jail. It is the grandfather of time series and predates influx by about a decade. It is still the fastest out there too. It is used by about every major financial and trading institution in the US and Europe. Everybody thinks TSDB are something new-ish, but they've been around since the days of APL. All you youngins disappointment me every time you write about time series, vector languages, or data-oriented programming and entirely neglect all the work that comes under the APL/Vector umbrella. SOA and DOD have been around for 50+ years, and they didn't start with c++ or Pandas. Now the creator, Arthur Whitney, has a new one out called Shakti that is even faster (but has also ditched from the "niceties" of Q. https://shakti.com/ https://shakti.com/
- kingforaday 2y agokdb is mentioned. Possible it was added after your comment in the last 30 mins, but the article is about open source not proprietary.
- jnordwick 2y agoI control-F'ed and didn't find it before I wrote that comment. I searched for KDB and APL, but found nothing. EDIT: yeah that sentence is new. I think some of the sentences around Influx being the "first" have been adjusted too now saying it is the first "mainstream" and called other db's niche. it's also a little off. KDB+ is more popular in large banks and trading firms than HFT places.
- deleted 2y ago[deleted]
- d0mine 2y agokdb is not open-source, is it? https://news.ycombinator.com/item?id=19973847 https://news.ycombinator.com/item?id=19973847
- ekianjo 2y agocorrect, it is not
- jmakov 2y agoWhy would you use this instead of CH if your usecase is metrics?
- dan-robertson 2y agoMore ergonomic queries, though less flexible than sql
- applied_heat 2y agoVm is really straightforward to get set up, I’ve heard clickhouse admin can be more involved
- candiddevmike 2y agoBecause there are multiple definitions of what a metrics usecase is.
- noselasd 2y agoWhat is "CH" in this context ?
- mthoms 2y agoClickHouse
- Mongoose 2y agoHad me until claiming that InfluxDB was the first mainstream TSDB in 2013. OpenTSDB (2010)? Graphite (2008)? RRDtool (1999)? Maybe Influx took off in a way these prior projects didn't, but people have been storing time series data for decades.
- PeterZaitsev 2y agoI think While OpenTSDB was reasonably general purpose, Graphite and RRDTools were done for very specific monitoring use cases.
- IOT_Apprentice 2y agoWe used a customized version of OpenTSDB at eBay to do monitoring of everything on the platform, including the search infrastructure. This was during the time eBay built a private cloud.
- johnorourke 2y agoRRDTool was the generalised version of the TSDB that was born for the specific use case, MRTG. It could be used for anything. Hopefully I've remembered that correctly!
- osigurdson 2y agoInfluxDB always seemed like it was run by children who are good at raising VC money.
- candiddevmike 2y agoEventually one of their rewrites will find product market fit.
- dijit 2y agoAgreed, after trying to buy a commercial license from them I was left… wanting to avoid the company entirely.
- steveBK123 2y ago
- suyash 2y agoInfluxDB is open source (https://github.com/influxdata https://github.com/influxdata) and seems to be leader by far as per DBEngines ranking : https://db-engines.com/en/ranking/time+series+dbms https://db-engines.com/en/ranking/time+series+dbms
- PeterZaitsev 2y agoHas things changed with InfluxDB 3.0 ? Previously InfluxDB was Open Core, with very crippled Community version (ie High Availability was Enterprise only)
- 00xnull 2y agoAWS interestingly launched a managed InfluxDB option under the “Timestream” product this year. I’ve personally found TimescaleDB to be a much easier to work with platform, even if there is more overhead in defining a schema and enabling compression, etc.
- lakomen 2y agoI've been using Influx v1 on a small scaleway instance for a while. I loved it and the tooling around it as well. I would get notifications on Telegram if events needing attention happened. But v3 is still under development so I'll explore Prometheus.
- dan-robertson 2y agoI wonder if the in-house metrics systems at big tech firms like Google and Facebook should be counted as ‘proprietary’ for these purposes. I suppose not because one can’t really pay to get them internally.
- mmooss 2y agoI might expect that for a specialized task, specialized database technology would outperform general relational databases. But what specific technologies have significant impacts on time series db efficiency?
- andreineculau 2y agoHow do these stack up to https://www.timescale.com/ https://www.timescale.com/ ?
- to11mtm 2y agoI haven't actually used Timescale but it was on my list to try after various gripes with how InfluxDB worked and limitations on tags. Broadly speaking I can say I'd trust TSDB over Influx for a lot of cases since it's using Postgres under the hood, and while I don't know precisely what issues we had adopting Influx, I know we had some as we 'scaled up' tho and that was when stuff like tag limits were brought up a lot (Interestingly, our app was 'barely passing' because we metric'd a lot but we also kept stuff just short enough for everyone to be OK with it, mostly because the metrics we got were very useful for things)
- deleted 2y ago[deleted]
- camel_gopher 2y agoWow this article misses many TSDBs that predated those listed. Victoria seems to be basically a rewrite of Prom.
- peteforde 2y agoIf the author ends up reading this... as someone who is ignorant but curious about time series databases, it was frustrating to see an example of what relational data looks like, but then no example of what time series data looks like. It might take a bit more context or setup, but this article is an opportunity to educate folks who aren't deep in the same trench. Edit: in fairness, this article doesn't describe itself as "a Postgres developer's guide to getting started with time series databases" and maybe that's what I was unfairly hoping I would find.
- osigurdson 2y agoThe naive relational time series table looks like: propertyid, timestamp, value Then add a covering index so your reads only ever hit the index. This works completely fine for low billions of rows. After that suggest using clickhouse. It is less general but at large enough scale you need to make some tradeoffs. Completely fine to start with a relational DB in many cases though.
- candiddevmike 2y agoBut then I have to use standard SQL instead of a vendor's half assed, poorly documented query language.
- peteforde 2y agoThanks for this. Can we keep going? The article mentioned stock prices, so let's use your schema: LLY.NYSE, 1726377148, 924.38 SHOP.NYSE, 1726377216, 72.45 SHOP.NYSE, 1726377245, 72.41 LLY.NYSE, 1726377284, 924.39 LLY.NYSE, 1726377310, 924.36 Okay, so you're appending values that capture a moment in time for a given index, with the understanding that you're almost never going to revise a tuple. So far as we're concerned, time only flows in one direction. Periodicity is determined by the client application but could be throttled due to a bottleneck, resulting in fewer writes. I can imagine how storing in blocks based on time ranges and the fact that time number goes up makes this very easy to index. With even this simple example, it's clear what this type of datastore could be useful for, even though I'm not convinced you couldn't do pretty exciting things in Redis or sqlite3. That said, I'm still significantly confused why the hell you can only store numeric values. It just seems like a very arbitrary constraint.
- mrgoldenbrown 2y agoThey are definitely using a more narrow definition of time series database than I expected - they don't mention rrd for example, which is GPL and been around for decades.
- nitinreddy88 2y agoCan anyone comment why these are superior to ClickHouse? I really like the ClickBench which compares the various products performance (and open source).
- alphabettsy 2y agoClickhouse is great, but it’s more complex. It’s not a direct comparison but it’s like Redis to Postgres.
- osigurdson 2y agoI have no affiliation with ClickHouse, but in my experience, everything I have tried (regular relational DB (Postgres), InfluxDB, TimescaleDB is significantly inferior to it. However, I wouldn't bother with it unless you have enough scale to justify it (imo that is > low billions of rows).
- PRANSHURAJPUT 2y ago[flagged]
- erbdex 2y agoGraphite[1] has interesting plugin ecosystem like Grafana[2] & has been around since 2009. Architecture of Open Source Applications book[3] has done a nice overview of the architecture. I really like how it reuses linux filesystem as data store, making archiving very simple. [1]: https://graphiteapp.org/ https://graphiteapp.org/ [2]: https://grafana.com/ https://grafana.com/ [3]: https://aosabook.org/en/v1/graphite.html https://aosabook.org/en/v1/graphite.html
- noselasd 2y agoMy biggest beef with VictoriaMetrics is I can't delete and re-ingest a range of data. i.e. if I discover yesterdays data was messed up due to some issue in downstream data processing, I can't delete yesterdays data and re-ingest the cleaned up data. (Or even just delete yesterdays bad data) The only workaround I've found so far is to dump out the whole timeseries which could go back months/years, delete the timeseries, fix/delete data in the dump and re-ingest the whole thing. This really, really sucks.
- applied_heat 2y agoI think you can delete just a timestamp range of a series I will look up the query and get back to you