13 ms·
Druid: fast column-oriented distributed data store
- sspring1 11y agoGreat, just what we need, a Druish database.
- deleted 11y ago[deleted]
- ciconia 11y agoFunny, it doesn't look druish...
- brightball 11y agoIt takes only what it needs to survive...
- brightball 11y agoWe're down voting Spaceballs references? Who are you monsters!?
- thecatspaw 11y agopeople who come here for serious discussions. Reddit is more the place for memes, references and stuff like that, not HN
- tschellenbach 11y agoDruid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.
- feral 11y agoSecond that experience: When I was at Optimizely, my team chose Druid for a large-scale analytics application, after a pretty extensive benchmarking. It was very impressive, though not trivial to set up. Blog post with a bit more detail: https://medium.com/engineers-optimizely/slicing-and-dicing-data-with-druid-f61cca8a63d2#.rqwqksjgt https://medium.com/engineers-optimizely/slicing-and-dicing-d...
- fangjin 11y agoDruid committer here. We spent a lot of time in the early days on making sure the system worked (at scale) and now we're spending more time to make it much easier to set up and manage.
- pookeh 11y agoIt might be helpful perhaps to provide some kubernetes configuration set up (or like a presetup kubernetes running in vagrant) that has all the nodes correctly configured out of the box to easily get started with development and prototyping.
- vog 11y agoI'm a bit surprised about forking the druid chef cookbook. From the blog post: > we were delighted to find an existing community cookbook, chef-druid, that can configure and deploy a druid cluster. However, that community cookbook hasn’t been updated for nearly a year and does not support the latest version of druid. We have therefore forked off our own version, optimizely/chef-druid, which supports the latest version of druid. Why didn't you create a pull request to the upstream project? That way, all people who are using the original cookbook could have directly benefit from your improvements, without having to discover the new fork.
- feral 11y agoI wasn't involved in that specific implementation decision, so can't say. Speculating personally, it was probably just something that happened for speed/ease - as opposed to trying to get an apparently abandoned repo rolling again. We did make optimizely/chef-druid open and public though, and while I'm not with Optimizely any more, I'm sure the good folk there would be more than happy to merge back into the original repo if anyone was still maintaining it and wanted that to happen.
- jnordwick 11y agoEvery open source column database I've seen is very poor: text, no decent array oriented ability (give me the prevoius row), slow, json output, etc. When will somebody get it right?
- ktamura 11y agoThis is simply false. PostgreSQL + cstore_ftw, Presto, Impala, etc. all support "array oriented ability" via window functions.
- jnordwick 11y agoI mean like give me the preceding row from the current. Very useful for calculating anything from a simple rate of change to a properly doing a temporal database. Arrays as a column type doesn't get you that.
- teraflop 11y agoNo, but window functions do, unless I'm not understanding what your goal is. http://www.postgresql.org/docs/current/static/functions-window.html http://www.postgresql.org/docs/current/static/functions-wind...
- jnordwick 11y agoI actually was using that yesterday. So so so so very very very very slow.
- slachterman 11y agoImpala supports LAG and LEAD, etc.: http://www.cloudera.com/documentation/archive/impala/2-x/2-0-x/topics/impala_analytic_functions.html http://www.cloudera.com/documentation/archive/impala/2-x/2-0...
- jnordwick 11y agoSo do most databased, but they are SLOOOOOOOW.
- rosalinekarr 11y agoYeah, more database solutions that's what we need. https://xkcd.com/927/ https://xkcd.com/927/
- Exuma 11y agoIt actually is needed. It's a great enterprise-level timeseries-centric database.
- whitegrape 11y agoHas anyone done a meaningful private benchmark comparison with http://www.scylladb.com/ http://www.scylladb.com/ ? I didn't find one online.
- ddorian43 11y agoIt is a 100% different type of database. Druid is olap while scylladb is oltp. They have nothing in common (except for the "columnar" name)
- whitegrape 11y agoYes, but that doesn't mean you can't benchmark them anyway. And I think you could probably find some meaningful comparison. Certainly it would be more useful than the Druid whitepaper's benchmarks against MySQL. (I used to work on a DB project, we too had a benchmark against MySQL even though our DB was OLAP-focused.)
- ddorian43 11y agoNo they can't because scylladb can't do the stuff druid can (and vice-versa). While mysql can (even though it's a completely different way).
- NamTaf 11y agoThe realtime ingestion is interesting especially if I can still batch import. When processing machine data, I've found that a quantity of sources come in chunks (logfiles written out every 24 hours for exmaple) but the eventual aim is to migrate to realtime (i.e.: a data point every n seconds/minutes/etc. where you instantly consume that data point) streaming. If this transition is easy without reworking infrastructure, the solution is far more attractive.
- techwizrd 11y agoA friend of mine who interned with me at eBay used Druid and Angular to great success to build a tool for analysts to look at trends in our data. Druid is some seriously cool stuff.
- whitenoice 11y agoHow does it compare to Vertica?
- mrweasel 11y agoWhat's the advantage of a "column-oriented" data store/database?
- gglanzani 11y agoAnalytics like sum(column) are much faster if data is saved column-oriented. Also every time you don't do select *, but select a, b, it's much easier on disk because you're not even touching some columns.
- scott_s 11y agoWikipedia is a good start: https://en.wikipedia.org/wiki/Column-oriented_DBMS https://en.wikipedia.org/wiki/Column-oriented_DBMS The references section has good primary sources.
- Exuma 11y agoOur 2-man team set up Druid........ i took 5+ months and was excruciating to configure and get running smoothly (things were slightly more complicated because we decided to use docker). It also took ~30 servers to make a truly fault-tolerant setup. With that said, it works very well, but it definitely came at the cost of a good dose of sanity.
- fangjin 11y agoFWIW, Imply repackaged Druid in such a way that should make it much much easier to set up and evaluate. We've been porting our docs over to Druid for 0.9.0: http://imply.io/docs/latest/quickstart http://imply.io/docs/latest/quickstart There's also a production-ready docker distribution: https://hub.docker.com/r/imply/imply/ https://hub.docker.com/r/imply/imply/