5 ms·

Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and

Druid is quickly becoming the leading open source solution for building highly scalable analytics. We evaluated it for getstream.io. Unfortunately the setup and maintenance is still very labour intensive. For startups that's a concern. Many larger companies we spoke to were extremely happy about running Druid in production though.

feral 11y ago
Second that experience: When I was at Optimizely, my team chose Druid for a large-scale analytics application, after a pretty extensive benchmarking. It was very impressive, though not trivial to set up. Blog post with a bit more detail: https://medium.com/engineers-optimizely/slicing-and-dicing-data-with-druid-f61cca8a63d2#.rqwqksjgt https://medium.com/engineers-optimizely/slicing-and-dicing-d...
- fangjin 11y ago
  Druid committer here. We spent a lot of time in the early days on making sure the system worked (at scale) and now we're spending more time to make it much easier to set up and manage.
  - pookeh 11y ago
    It might be helpful perhaps to provide some kubernetes configuration set up (or like a presetup kubernetes running in vagrant) that has all the nodes correctly configured out of the box to easily get started with development and prototyping.
- vog 11y ago
  I'm a bit surprised about forking the druid chef cookbook. From the blog post: > we were delighted to find an existing community cookbook, chef-druid, that can configure and deploy a druid cluster. However, that community cookbook hasn’t been updated for nearly a year and does not support the latest version of druid. We have therefore forked off our own version, optimizely/chef-druid, which supports the latest version of druid. Why didn't you create a pull request to the upstream project? That way, all people who are using the original cookbook could have directly benefit from your improvements, without having to discover the new fork.
  - feral 11y ago
    I wasn't involved in that specific implementation decision, so can't say. Speculating personally, it was probably just something that happened for speed/ease - as opposed to trying to get an apparently abandoned repo rolling again. We did make optimizely/chef-druid open and public though, and while I'm not with Optimizely any more, I'm sure the good folk there would be more than happy to merge back into the original repo if anyone was still maintaining it and wanted that to happen.
threeseed 11y ago
Druid isn't even close to being the leading solution. The overwhelming majority of places doing big data analytics will be Hadoop/Spark using core HDFS. Then most places will augment this with a range of database solutions depending on how structured/clean the data is and the various workloads. Cassandra, HBase, MongoDB, Teradata, Oracle, ElasticSearch, Greenplum are all pretty common place in most enterprises. And of course the ridiculous array of SQL engines on top of HDFS/S3/whatever else e.g. Hive, Spark SQL, Presto, Drill, SAP etc.
- fangjin 11y ago
  Druid's main value add to the data infrastructure space is around power user-facing data applications at scale. The queries it is best at are OLAP/business intelligence style queries. It isn't really designed to be a general processing tool such as Hadoop or Spark. The open source data space is very complex, and there are many different solutions targeted towards many different use cases. Druid is better than other solutions at some of these use cases, and worse than other solutions at others. I wrote my interpretation of the current open source data landscape here for anyone interested: http://imply.io/post/2015/11/04/big-data-zoo.html http://imply.io/post/2015/11/04/big-data-zoo.html
  - deepanchor 11y ago
    As someone who hasn't yet had the opportunity to use many of these systems, this was a great high-level overview of how the various systems fit together. Thanks for writing it!
  - scott_s 11y ago
    Your Wikipedia link to "stream processors" is for the wrong kind of stream processors. For a decades, the digital signal processing and graphics worlds have used the streaming abstraction to design programs and hardware. Their applications are typically expressed in pipelines, and do have to continually process data. "Big data" stream processing is obviously related as its a dataflow programming model, but it's still very different in practice. The streaming abstraction is generally more free-form, and not realized directly in hardware. I contrast the two kinds of streaming in Section 2 of a paper from a few years ago: http://www.scott-a-s.com/files/pact2012.pdf http://www.scott-a-s.com/files/pact2012.pdf
eva1984 11y ago
No it isn't Elasticsearch and Redshift are.
- ci5er 11y ago
  Huh? GP will make an unsubstantiated claim and you'll double down with an unsubstantiated counter claim (in one line!) Seriously: What, specifically, do you disagree with, in the context of your use case, and what is your use-case?
- Exuma 11y ago
  That doesn't make any sense at all.
RyanHamilton 11y ago
"becoming the leading open source solution" is something you can say that doesn't mean much and is difficult to prove or disprove. There are a number of entrenched players, open source and commercial that provide a column oriented database: http://www.timestored.com/time-series-data/column-oriented-databases http://www.timestored.com/time-series-data/column-oriented-d... That's a short description of 12 alternatives I wrote up. Druid does look interesting, it appears to have got the architecture pretty spot on and I'm going to look into it more.
mau 11y ago
Much of the effort for the next release of Druid (0.9) has been put into docs. Even with that you still need to handle a complex cluster made of several services: zookeeper, druid overlord, druid middlemanager, druid broker, etc... still it sounds more complicate than it actually is. What was the alternative solution adopted by your company?

feral 11y ago

Second that experience: When I was at Optimizely, my team chose Druid for a large-scale analytics application, after a pretty extensive benchmarking. It was very impressive, though not trivial to set up. Blog post with a bit more detail: https://medium.com/engineers-optimizely/slicing-and-dicing-data-with-druid-f61cca8a63d2#.rqwqksjgt https://medium.com/engineers-optimizely/slicing-and-dicing-d...

fangjin 11y ago
Druid committer here. We spent a lot of time in the early days on making sure the system worked (at scale) and now we're spending more time to make it much easier to set up and manage.
- pookeh 11y ago
  It might be helpful perhaps to provide some kubernetes configuration set up (or like a presetup kubernetes running in vagrant) that has all the nodes correctly configured out of the box to easily get started with development and prototyping.
vog 11y ago
I'm a bit surprised about forking the druid chef cookbook. From the blog post: > we were delighted to find an existing community cookbook, chef-druid, that can configure and deploy a druid cluster. However, that community cookbook hasn’t been updated for nearly a year and does not support the latest version of druid. We have therefore forked off our own version, optimizely/chef-druid, which supports the latest version of druid. Why didn't you create a pull request to the upstream project? That way, all people who are using the original cookbook could have directly benefit from your improvements, without having to discover the new fork.
- feral 11y ago
  I wasn't involved in that specific implementation decision, so can't say. Speculating personally, it was probably just something that happened for speed/ease - as opposed to trying to get an apparently abandoned repo rolling again. We did make optimizely/chef-druid open and public though, and while I'm not with Optimizely any more, I'm sure the good folk there would be more than happy to merge back into the original repo if anyone was still maintaining it and wanted that to happen.

threeseed 11y ago

Druid isn't even close to being the leading solution. The overwhelming majority of places doing big data analytics will be Hadoop/Spark using core HDFS. Then most places will augment this with a range of database solutions depending on how structured/clean the data is and the various workloads. Cassandra, HBase, MongoDB, Teradata, Oracle, ElasticSearch, Greenplum are all pretty common place in most enterprises. And of course the ridiculous array of SQL engines on top of HDFS/S3/whatever else e.g. Hive, Spark SQL, Presto, Drill, SAP etc.

fangjin 11y ago
Druid's main value add to the data infrastructure space is around power user-facing data applications at scale. The queries it is best at are OLAP/business intelligence style queries. It isn't really designed to be a general processing tool such as Hadoop or Spark. The open source data space is very complex, and there are many different solutions targeted towards many different use cases. Druid is better than other solutions at some of these use cases, and worse than other solutions at others. I wrote my interpretation of the current open source data landscape here for anyone interested: http://imply.io/post/2015/11/04/big-data-zoo.html http://imply.io/post/2015/11/04/big-data-zoo.html
- deepanchor 11y ago
  As someone who hasn't yet had the opportunity to use many of these systems, this was a great high-level overview of how the various systems fit together. Thanks for writing it!
- scott_s 11y ago
  Your Wikipedia link to "stream processors" is for the wrong kind of stream processors. For a decades, the digital signal processing and graphics worlds have used the streaming abstraction to design programs and hardware. Their applications are typically expressed in pipelines, and do have to continually process data. "Big data" stream processing is obviously related as its a dataflow programming model, but it's still very different in practice. The streaming abstraction is generally more free-form, and not realized directly in hardware. I contrast the two kinds of streaming in Section 2 of a paper from a few years ago: http://www.scott-a-s.com/files/pact2012.pdf http://www.scott-a-s.com/files/pact2012.pdf

eva1984 11y ago

No it isn't Elasticsearch and Redshift are.

ci5er 11y ago
Huh? GP will make an unsubstantiated claim and you'll double down with an unsubstantiated counter claim (in one line!) Seriously: What, specifically, do you disagree with, in the context of your use case, and what is your use-case?
Exuma 11y ago
That doesn't make any sense at all.

RyanHamilton 11y ago

"becoming the leading open source solution" is something you can say that doesn't mean much and is difficult to prove or disprove. There are a number of entrenched players, open source and commercial that provide a column oriented database: http://www.timestored.com/time-series-data/column-oriented-databases http://www.timestored.com/time-series-data/column-oriented-d... That's a short description of 12 alternatives I wrote up. Druid does look interesting, it appears to have got the architecture pretty spot on and I'm going to look into it more.

mau 11y ago

Much of the effort for the next release of Druid (0.9) has been put into docs. Even with that you still need to handle a complex cluster made of several services: zookeeper, druid overlord, druid middlemanager, druid broker, etc... still it sounds more complicate than it actually is. What was the alternative solution adopted by your company?