3 ms·

There is no "the" data stack. Some organizations have complicated orchestration requirements, some don't. Some organizations have large amounts of heterogeneou

There is no "the" data stack. Some organizations have complicated orchestration requirements, some don't. Some organizations have large amounts of heterogeneous data, some don't. Some organizations need to run large distributed machine learning tasks, some don't. If you adopt tools that don't actually solve real problems that your team/org faces, then of course you'll end up with a bunch of extraneous fluff. Your comment has the same flavor as the frequent HN meme of "I just use Vi and Tmux and a BSD server for my business and anything else is a sign of organizational failure and individual weakness". All of these tools exist for a reason. Where you get into trouble is losing focus on solving your real practical problems.

Lyngbakr 3y ago
> Where you get into trouble is losing focus on solving your real practical problems. Absolutely agree. As I say YMMV, but my experience is that often people have adopted tools because "that's what data teams use" rather than because it's the best tool for the job. Often, we could — and should — build an in-house solution that fits perfectly rather than buying (every month) an imperfect solution.
CRConrad 3y ago
> Some organizations need to run large distributed machine learning tasks, some don't. I think almost no organizations need to run "large distributed machine learning tasks". Many just think they do, because it's the buzzword of the moment.
- nerdponx 3y ago
  Also true, but I think maybe that's a great example of what I and OP are saying here. Do you really need it? Or do you just think you might need it one day because people are talking about it on Medium, or because you had a nice chat with some sales rep? Or do you just want it on your resume?
- disgruntledphd2 3y ago
  Facebook's ad prediction models weren't distributed until maybe 2014 or 2015. Just buying bigger boxes can get you a loooooinnnng way.

Lyngbakr 3y ago

> Where you get into trouble is losing focus on solving your real practical problems. Absolutely agree. As I say YMMV, but my experience is that often people have adopted tools because "that's what data teams use" rather than because it's the best tool for the job. Often, we could — and should — build an in-house solution that fits perfectly rather than buying (every month) an imperfect solution.

CRConrad 3y ago

> Some organizations need to run large distributed machine learning tasks, some don't. I think almost no organizations need to run "large distributed machine learning tasks". Many just think they do, because it's the buzzword of the moment.

nerdponx 3y ago
Also true, but I think maybe that's a great example of what I and OP are saying here. Do you really need it? Or do you just think you might need it one day because people are talking about it on Medium, or because you had a nice chat with some sales rep? Or do you just want it on your resume?
disgruntledphd2 3y ago
Facebook's ad prediction models weren't distributed until maybe 2014 or 2015. Just buying bigger boxes can get you a loooooinnnng way.