8 ms·
A Review of Time Series Databases
- fnl 10y agoWhat about BTrDB from Berkeley? http://btrdb.io/ http://btrdb.io/
- dataloopio 10y agoIt's in the spreadsheet. I've not tried it and there wasn't much info available at the time of writing the blog.
- TheHippo 10y agoDataloop develops DalmatinerDB. Guess which database won first place.
- jp3141 10y ago> Disclaimer: Dataloop helps build DalmatinerDB so there is a massive conflict of interest.
- marianoguerra 10y agoit's developed by @heinz_gies from Project FiFo, they may pay him to do changes on it, but it wasn't started there and AFAIK heinz doesn't work there full time.
- Licenser 10y agoMariano is correct, DalmatinerDB is build and maintained by Project-FiFo. Dataloop (the authors of the blog post) is currently the biggest user (as far as I know at least), using it in their SaaS. They have however been excellent open source citizens and contributed back improvements, bug reports and suggestions.
- dintech 10y agoKDB+ is the number 1 time-series database, no question. Per-core, it's 10 times faster than any other time-series database. Seems like an odd omission. http://kparc.com/q4/readme.txt http://kparc.com/q4/readme.txt
- wlesieutre 10y agoAs noted in the scope section along with some other omissions: > Only free and open source time series databases and their features have been compared. Therefore if someone asks “have you tried Kdb+ or Informix?” the answer will be no. They are probably awesome though. Article could have been titled "Review of the Top10 FOSS Time Series Databases" since that's what it is.
- gaius 10y agoThe 32-bit version is free. I agree that any comparison skipping it is meaningless, it is the "standard" in this space, to the extent there is one.
- evanpw 10y ago> The 32-bit version is free. Only for non-commercial use: https://kx.com/2015/09/19/32-bit-kdb-for-non-commercial-use-only/ https://kx.com/2015/09/19/32-bit-kdb-for-non-commercial-use-...
- threeseed 10y agoKDB+ does have some disadvantages though. Requiring a schema makes it ill-suited to many use cases since in many situations you don't know the schema ahead of time. And the biggest for me is poor Hadoop/Spark integration. JDBC is not a great choice since you have poor predicate pushdown support and it prevents say Spark from going specifically to the node with the data.
- gaius 10y agoYou always have a schema. The only choice you have is whether your tool helps you validate it or not.
- triplesec 10y agoTo be more precise, this is a review of more than 10 free and open source time series databases, wherein the author chooses 10 favourites, by some metrics.
- threeseed 10y agoWhat do you think is missing ? Because those are probably the 10 most popular and well known.
- polyfractal 10y agoWell, for one thing, this isn't even attempting an objective comparison. The article basically grabs whatever third party benchmark numbers are available and pretends all the metrics are comparable. In many cases, the external benchmarks are testing entirely different things, so it's meaningless to give those numbers in a "review" of all the options. It's really just a "top ten list according to my arbitrary thoughts", which is fine. But it's not really a useful comparison at all.
- triplesec 10y agoI have no idea; it's not an area of my expertise. However, the author tests 16 and selects 10 by the methods shown. I'm glad for the reinforcement that these are likely reasonably decent ones by social proof via you!
- deleted 10y ago[deleted]
- sixhobbits 10y ago> I decided to pen a magnum opus of my own opinions "magnum opus (noun) a large and important work of art, music, or literature, especially one regarded as the most important work of an artist or writer." Considering that this review has a word count of 3708, describing it as a magnum opus (something I associate with a work that takes close to a lifetime to complete) rankles.
- 64bitter 10y agoI only counted 3705 words. I might need to count them again, damn fingers. Did you include the comments? I didn't myself, but thats sure to bump up the count. We should investigate what the cutoff limit is, maybe 10,000 words would qualify? I'm struggling weather to classify the work as art or literature, maybe both? Its definitely a seminal piece.
- alex_hitchins 10y agoYes, I feel the same. Akin to becoming a PhD by scanning the spines of books in a library. It's by no means bad work at all, but not a Magnum Opus.
- skrebbel 10y agoDear HN, recently time series DBs appear to have become very popular on HN but I'm confused. What are they for? I understand the idea for, say, physics experiments. Lots of parameters sampled thousands of times per second, gotta store that stuff somewhere. But this is HN, not an experimental physics subreddit, so I must be missing something. What do people on HN who care about this stuff use time series databases for? Why are there 20+ competitors?
- leetbulb 10y agoDruid provides realtime ad-hoc campaign analytics to our advertisers and publishers for our performance CPA network.
- dsl 10y agoAs an example, I track requests per second per datacenter in a time series database. This allows me to make pretty graphs and do traffic engineering between sites.
- majke 10y agoWhen you run servers, you might want to record plenty of parameters. From received packets counters, via disk io operations, to cpu load. If you want to identify issues with good precision, you will record _hundreds_ of counters per each server. Sticking this kind of data to relational database is not good. With couple of servers you might have millions of _writes_ per second. While the data is only read when you refresh some charts. This makes the time series db unique - they need to handle write-heavy loads, and perform well when asked to retrieve couple of parameters within defined time window. So two things: - time series db's are needed for anyone running live systems and measuring them on-the-fly - time series db's have unique access patterns, not usually optimized by more conventional databases
- paukiatwee 10y agoTime serie DB for storing: Monitoring data, IoT data, etc. See example supported inputs by Telegraf https://github.com/influxdata/telegraf/tree/master/plugins/inputs https://github.com/influxdata/telegraf/tree/master/plugins/i... from InfluxData
- socmag 10y agoI'm on my phone and my eyes are not so good as it is. Does he say what the architecture of the platform he was testing this on?
- socmag 10y agoNever mind, I found it in the Gist... https://gist.github.com/sacreman/b77eb561270e19ca973dd5055270fb28 https://gist.github.com/sacreman/b77eb561270e19ca973dd505527... 1 x DalmatinerBD server GCE n1-standard-16 (16 cpu, 60GB memory, 1 x 375G local SSD disk) I would assume that is the same setup they used for all the other tests, though not clear about that yet Anyone know the network bandwidth on these instance types?
- dataloopio 10y agoHi, I'm the original author. I think that GCE instance has 2Gb per core but I'd need to check. It wasn't bottlenecked on network bandwidth though. The other benchmarks are linked in the spreadsheet to their respective details. It's not an absolute, direct, fair comparison. However, we wanted to start somewhere with information available right now and try to collect better results over time as the respective interested parties benchmarked and blogged about their databases. I'd like to spend more time benchmarking every database in the spreadsheet but it feels like something the project owners should do themselves. I'd probably only get the setup wrong.
- socmag 10y agoThanks for the clarification, good to know.
- deleted 10y ago[deleted]
- bandris 10y agoThe image embedded in the Gist says 10Gb network.
- lormayna 10y agoWhat is a good TDB for collecting data from home automation sensors and that can run on a little PC like a Raspberry?
- threeseed 10y agoInfluxDB is perfect for these use cases. It's a single Go binary with excellent performance and use of resources. http://www.aymerick.com/2015/10/07/influxdb-telegraf-grafana-raspberry-pi.html http://www.aymerick.com/2015/10/07/influxdb-telegraf-grafana...
- xmatos 10y agoWhat specific features a time series database offer over mongo or redis?
- gaius 10y agoWell one advantage compared to Mongo is they actually keep your data safe.
- detaro 10y agoI wouldn't trust on all of them to do that reliably. If you look at a large group of database systems, some of them are going to have bugs or bad defaults as well.
- xmatos 10y agoreally? and how's that? to my understanding, data integrity is not a feature on any of them. but I asked that because I've been using redis a lot lately. I built a real-time analytics using bitmaps and I could use sorted sets to store time series data. I really wanted to know what benefits do they offer to justify adding something else to the stack...
- RyanHamilton 10y agoFor a similar comparison but that encompasses commercial databases as well see here: http://www.timestored.com/time-series-data/column-oriented-databases http://www.timestored.com/time-series-data/column-oriented-d...
- vslira 10y agoQuestion: Does it make sense to use a time-series database if you're not too sure that you can trust the "time" in the series? For example, let's say you're logging user's locations on a jogging app and using the system's clock as the time record (I understand this may not be ideal). Someone could log a run 2 years from now, for example.
- chaotic-good 10y agoIt is usually a good idea to discard late writes. It should be possible to write data with some time-stamp skew but if skew is too large data should be discarded as erroneous. It's totally depends on an application so.
- coredog64 10y ago> It seems there is a never ending supply of time series databases being written on top of Cassandra. Unless any of them are significantly better than KairosDB I'm not going to change the top 10. I really appreciate the work Brian Hawkins has done with KairosDB. We settled on KairosDB at work after having a bunch of trouble with InfluxDB. At the same time, it's very much stuck in the Cassandra 2.1 world* . There have been some significant advancements with Cassandra for this particular use case (new storage engine, TWCS) and KairosDB is missing out. *Yes, it can be run on 2.2.x by enabling the now-disabled-by-default Thrift, but that feels like a last gasp.
- JamesBarney 10y agoI'm evaluating InfluxDB for work and was wondering what issues you ran into?
- Licenser 10y agoBlind guess, lack of clustering and a history of changing file formats so you loose your data every few month.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]