4 ms·

Big Data Analytics Systems: What Goes Around Comes Around

32 points by mrry 11y ago

amit_m 11y ago
What are people actually using these days for analytics? What is missing? (i.e. what would the dream startup do for you?) There's a lot of hype around certain technologies, and I suspect it does not reflect reality. I've used hadoop+pig a few years ago and it was absolutely terrible.
- bagels 11y ago
  I'm working on moving some analytics from pig + hadoop streaming over to Redshift. The debug cycles are so much faster with sql queries.
  - blumkvist 11y ago
    You can use sql on hadoop...
  - threeseed 11y ago
    I am assuming you tried Hive, Impala, Spark SQL ? Redshift seems like a great fit for Adhoc analytics. Just a shame you can't run it in your own data centre (some of us can't use the cloud).
    - ddorian43 11y ago
      You can use citusdb (postgresql, columnar). Not the same, but close.
  - alecco 11y ago
    If you are using SQL... why didn't you try a columnar engine?
    - saosebastiao 11y ago
      Redshift is a columnar engine
  - bagels 11y ago
    To all those who have asked, yes, hive is also in use.
- threeseed 11y ago
  The majority of big data analytics is using the standard Hadoop stack and testing Spark. Dream startup would be to make the day to day life of running a cluster more manageable. If I had an app that could tell me how to optimize queries, tune the parameters and make the what/why/where/how questions easier I would buy it instantly.
  - virmundi 11y ago
    Look no further. Netezza by IBM dies that all with SQL. Seriously, I'm impressed by that product. It also supports Hadoop.
  - IndianAstronaut 11y ago
    I absolutely love Spark and really look forward to SparkR joining the main project this summer.
- alecco 11y ago
  If the data is structured, columnar databases still rule. But if you are stuck in SV's echo chamber it's all MapReduce/Hadoop/Spark/etc.
  - pacala 11y ago
    How about Spark + Parquet via SQLContext / SchemaRDD?
    - alecco 11y ago
      The overhead of that must be ridiculous.
      - pacala 11y ago
        What kind of overheads should we be aware of? Spark can run on a single machine, it could be an interesting benchmark to compare against Vertica or similar products. I've seen similar systems tuned for under 1s response times, even when running in distributed mode, though they were proprietary.
jacques_chester 11y ago
On slide 58, it's worth noting that Greenplum doesn't belong to EMC any more. It's been spun into Pivotal's Big Data Suite[0]. Disclaimer: I work for Pivotal Labs, another part of the same company. [0] http://pivotal.io/big-data/pivotal-big-data-suite http://pivotal.io/big-data/pivotal-big-data-suite
frankmcsherry 11y ago
Oh, it's coming around, alright. https://www.usenix.org/conference/hotos15/workshop-program/presentation/mcsherry https://www.usenix.org/conference/hotos15/workshop-program/p...
crimsonalucard 11y ago
Wrong. Big Data is not some analytics system. It's a method to cure cancer, stop terrorism and end world hunger. Honestly if you google "big data" + "cure cancer" or any of the things above you'll get an article about it. Big Data is over-hyped.

32 points by mrry 11y ago

What are people actually using these days for analytics? What is missing? (i.e. what would the dream startup do for you?) There's a lot of hype around certain technologies, and I suspect it does not reflect reality. I've used hadoop+pig a few years ago and it was absolutely terrible.

bagels 11y ago
I'm working on moving some analytics from pig + hadoop streaming over to Redshift. The debug cycles are so much faster with sql queries.
- blumkvist 11y ago
  You can use sql on hadoop...
- threeseed 11y ago
  I am assuming you tried Hive, Impala, Spark SQL ? Redshift seems like a great fit for Adhoc analytics. Just a shame you can't run it in your own data centre (some of us can't use the cloud).
  - ddorian43 11y ago
    You can use citusdb (postgresql, columnar). Not the same, but close.
- alecco 11y ago
  If you are using SQL... why didn't you try a columnar engine?
  - saosebastiao 11y ago
    Redshift is a columnar engine
- bagels 11y ago
  To all those who have asked, yes, hive is also in use.
threeseed 11y ago
The majority of big data analytics is using the standard Hadoop stack and testing Spark. Dream startup would be to make the day to day life of running a cluster more manageable. If I had an app that could tell me how to optimize queries, tune the parameters and make the what/why/where/how questions easier I would buy it instantly.
- virmundi 11y ago
  Look no further. Netezza by IBM dies that all with SQL. Seriously, I'm impressed by that product. It also supports Hadoop.
- IndianAstronaut 11y ago
  I absolutely love Spark and really look forward to SparkR joining the main project this summer.
alecco 11y ago
If the data is structured, columnar databases still rule. But if you are stuck in SV's echo chamber it's all MapReduce/Hadoop/Spark/etc.
- pacala 11y ago
  How about Spark + Parquet via SQLContext / SchemaRDD?
  - alecco 11y ago
    The overhead of that must be ridiculous.
    - pacala 11y ago
      What kind of overheads should we be aware of? Spark can run on a single machine, it could be an interesting benchmark to compare against Vertica or similar products. I've seen similar systems tuned for under 1s response times, even when running in distributed mode, though they were proprietary.

jacques_chester 11y ago

On slide 58, it's worth noting that Greenplum doesn't belong to EMC any more. It's been spun into Pivotal's Big Data Suite[0]. Disclaimer: I work for Pivotal Labs, another part of the same company. [0] http://pivotal.io/big-data/pivotal-big-data-suite http://pivotal.io/big-data/pivotal-big-data-suite

frankmcsherry 11y ago

Oh, it's coming around, alright. https://www.usenix.org/conference/hotos15/workshop-program/presentation/mcsherry https://www.usenix.org/conference/hotos15/workshop-program/p...

crimsonalucard 11y ago

Wrong. Big Data is not some analytics system. It's a method to cure cancer, stop terrorism and end world hunger. Honestly if you google "big data" + "cure cancer" or any of the things above you'll get an article about it. Big Data is over-hyped.