4 ms·

My recommendations... For automated cluster building - https://ambari.apache.org/ https://ambari.apache.org/ For analysing your data, dynamically building que

My recommendations... For automated cluster building - https://ambari.apache.org/ https://ambari.apache.org/ For analysing your data, dynamically building queries and sharing this with other people in your company - https://zeppelin.incubator.apache.org/ https://zeppelin.incubator.apache.org/ And coming soon - https://www.zeppelinhub.com/ https://www.zeppelinhub.com/

TallGuyShort 11y ago
I'm especially excited about Zeppelin. Using IPython for SciPy and smaller datasets is great. I would love it for the big data space I work in and Python's tooling to come together more.
- nl 11y ago
  IPython/Juypter works well against Spark. We have it working in production like that, and both Google[1] and IBM[2] do the same. [1] https://cloud.google.com/datalab/overview https://cloud.google.com/datalab/overview [2] https://www.ng.bluemix.net/docs/services/AnalyticsforApacheSpark/index.html https://www.ng.bluemix.net/docs/services/AnalyticsforApacheS...
- henridf 11y ago
  I recently looked into notebooks and found Beaker (http://beakernotebook.com http://beakernotebook.com) to be especially interesting in its support for passing data across languages.
nl 11y ago
I looked pretty hard at Zeppelin around 6 months ago, comparing it to iPython/Jupyter for use with Spark. I found Zeppelin hard to install (I'm a Java programmer and Zeppelin is in Scala/Java so I expected the opposite). It was also extremely buggy. Jupyter OTOH worked straight away, and even getting Spark integration working was straight forward compared to getting Zeppelin just working. Zeppelin looks nicer, and some of the features look great. It just isn't there for production use atm though.
clebio 11y ago
What about for provisioning clusters that don't require Hadoop. I suppose this could be akin to the comment about Redis -- we're working on deploying Kafka, Storm, and Zookeeper (none of which need Hadoop), and provisioning and node management (membership, leader election) in a dynamic environment (e.g. AWS autoscaling) is not at all obvious. There's also a paucity of substantive information about scaling these clusters dynamically.
lars_francke 11y ago
If you haven't had the chance yet I suggest you try Cloudera Manager & CDH instead of Ambari. I use both with clients and CM is years ahead of Ambari in terms of functionality and stability.
nisa 11y ago
Ambari looks nice from the outside but it's some kind of zombie that is full of os-specific stuff, opaque puppet stuff, python, java and even brings nagios and ganglia to the party. If it works it's probably fine, have fun debugging that stuff through. We are happy with https://github.com/saltstack-formulas/hadoop-formula https://github.com/saltstack-formulas/hadoop-formula and PXE booting an image. You can configure all aspects out of a single pillar file. Still full of warts but at least you have full control.

TallGuyShort 11y ago

I'm especially excited about Zeppelin. Using IPython for SciPy and smaller datasets is great. I would love it for the big data space I work in and Python's tooling to come together more.

nl 11y ago
IPython/Juypter works well against Spark. We have it working in production like that, and both Google[1] and IBM[2] do the same. [1] https://cloud.google.com/datalab/overview https://cloud.google.com/datalab/overview [2] https://www.ng.bluemix.net/docs/services/AnalyticsforApacheSpark/index.html https://www.ng.bluemix.net/docs/services/AnalyticsforApacheS...
henridf 11y ago
I recently looked into notebooks and found Beaker (http://beakernotebook.com http://beakernotebook.com) to be especially interesting in its support for passing data across languages.

nl 11y ago

I looked pretty hard at Zeppelin around 6 months ago, comparing it to iPython/Jupyter for use with Spark. I found Zeppelin hard to install (I'm a Java programmer and Zeppelin is in Scala/Java so I expected the opposite). It was also extremely buggy. Jupyter OTOH worked straight away, and even getting Spark integration working was straight forward compared to getting Zeppelin just working. Zeppelin looks nicer, and some of the features look great. It just isn't there for production use atm though.

clebio 11y ago

What about for provisioning clusters that don't require Hadoop. I suppose this could be akin to the comment about Redis -- we're working on deploying Kafka, Storm, and Zookeeper (none of which need Hadoop), and provisioning and node management (membership, leader election) in a dynamic environment (e.g. AWS autoscaling) is not at all obvious. There's also a paucity of substantive information about scaling these clusters dynamically.

lars_francke 11y ago

If you haven't had the chance yet I suggest you try Cloudera Manager & CDH instead of Ambari. I use both with clients and CM is years ahead of Ambari in terms of functionality and stability.

nisa 11y ago

Ambari looks nice from the outside but it's some kind of zombie that is full of os-specific stuff, opaque puppet stuff, python, java and even brings nagios and ganglia to the party. If it works it's probably fine, have fun debugging that stuff through. We are happy with https://github.com/saltstack-formulas/hadoop-formula https://github.com/saltstack-formulas/hadoop-formula and PXE booting an image. You can configure all aspects out of a single pillar file. Still full of warts but at least you have full control.