4 ms·

This is a very good point. I think many people are so caught up in bashing the hype train around big data and data science that they just casually dismiss thes

by ende 10y ago

This is a very good point. I think many people are so caught up in bashing the hype train around big data and data science that they just casually dismiss these incredibly valid points. It's not necessarily about how big your data is right now, but how big your data will be in the very near future. Retooling then is often a lot more difficult than just tooling properly up front, so even if some Spark operations might seem to add unnecessary overhead right now the lower transition cost down the road is often worth it.

jacques_chester 10y ago
Day 2 problems don't come to mind during days 0 and 1.
- shitgoose 10y ago
  Sometimes they don't come at all.
  - shitgoose 10y ago
    That was quote from Austin Powers. And a reference to the fact that Day 2 requirements almost never come.
elcritch 10y ago
These are valid points, and I agree many underestimate the cost of retooling and infrastructure. However, I am working on a team of smart engineers, but shell scripting is new to them, much less learning a full Hadoop / spark setup and associated tools. Luckily, you can often have your cake and eat it too: https://apidocs.joyent.com/manta/example-line-count-by-extension.html https://apidocs.joyent.com/manta/example-line-count-by-exten... Super useful system so far, and my goal is to allow our team to learn some basic scripting techniques and then run them on our internal cloud using almost identical tooling. Plus things like simple Python scripts are really easy to teach, and with this infrastructure it can scale quickly!
xorgar831 10y ago
I think the point is if you really have big data then it makes sense, but many shops add huge cost and complexity to projects where simpler tools would be more than adequate.
- ende 10y ago
  That's definitely true too. Being able to accurately assess whether that need will ever exist (or not) early on is invaluable.
- luckydata 10y ago
  It's the tooling, not the size of the data. Using "big data ecosystem" tools allows you to use all kinds of useful things like Airflow for pipeline processing, Presto to query the data, Spark for enrichment and machine learning etc... all of that without moving the data, which simplifies greatly metadata management which has to be done if you're serious about things like data provenance and quality.
  - qwertyuiop924 10y ago
    ...and how often do you need to do all that to a dataset?
  - xorgar831 10y ago
    A SQL db + Tableau are vastly more powerful and mature than those tools, they just can't do "big data", that's all.
    - luckydata 10y ago
      the data preparation work involved in doing anything trivial with that setup is mind boggling. I'm going to assume that this very cavalier assumption comes from a place of relatively little knowledge. Do some of this and you'll realize very soon you'd rather shoot yourself than doing complex data prep on constantly evolving schema using schema-on-write tools.

by ende 10y ago

jacques_chester 10y ago

Day 2 problems don't come to mind during days 0 and 1.

shitgoose 10y ago
Sometimes they don't come at all.
- shitgoose 10y ago
  That was quote from Austin Powers. And a reference to the fact that Day 2 requirements almost never come.

elcritch 10y ago

These are valid points, and I agree many underestimate the cost of retooling and infrastructure. However, I am working on a team of smart engineers, but shell scripting is new to them, much less learning a full Hadoop / spark setup and associated tools. Luckily, you can often have your cake and eat it too: https://apidocs.joyent.com/manta/example-line-count-by-extension.html https://apidocs.joyent.com/manta/example-line-count-by-exten... Super useful system so far, and my goal is to allow our team to learn some basic scripting techniques and then run them on our internal cloud using almost identical tooling. Plus things like simple Python scripts are really easy to teach, and with this infrastructure it can scale quickly!

xorgar831 10y ago

I think the point is if you really have big data then it makes sense, but many shops add huge cost and complexity to projects where simpler tools would be more than adequate.

ende 10y ago
That's definitely true too. Being able to accurately assess whether that need will ever exist (or not) early on is invaluable.
luckydata 10y ago
It's the tooling, not the size of the data. Using "big data ecosystem" tools allows you to use all kinds of useful things like Airflow for pipeline processing, Presto to query the data, Spark for enrichment and machine learning etc... all of that without moving the data, which simplifies greatly metadata management which has to be done if you're serious about things like data provenance and quality.
- qwertyuiop924 10y ago
  ...and how often do you need to do all that to a dataset?
- xorgar831 10y ago
  A SQL db + Tableau are vastly more powerful and mature than those tools, they just can't do "big data", that's all.
  - luckydata 10y ago
    the data preparation work involved in doing anything trivial with that setup is mind boggling. I'm going to assume that this very cavalier assumption comes from a place of relatively little knowledge. Do some of this and you'll realize very soon you'd rather shoot yourself than doing complex data prep on constantly evolving schema using schema-on-write tools.