3 ms·

Meh, depends on the data teams (and their chosen suite of tools)... Or on the Excel users Both sides can either hack together horrible work-arounds (a matter o

by b-luu 4y ago

Meh, depends on the data teams (and their chosen suite of tools)... Or on the Excel users Both sides can either hack together horrible work-arounds (a matter of "when you have a hammer, everything looks like a nail...") as well as brilliantly thought through solutions. Each tool should be used for it's best use cases, but not bent into what it wasn't designed for! IMHO spreadsheets excel at intuitively manipulating the data ON the data itself. While "modern data" tools (especially dbt) try to convert date teams to use developer best practices... At the expense of less intuitive/direct manipulation of the data. That being said, I think there are also things we could explore in that space: how to make the modern data stack more intuitive?

datalopers 4y ago
> how to make the modern data stack more intuitive? I'd start with getting people to learn relational data modeling and SQL [1][2] at a deep level. Stop reaching for python and pandas/spark for every basic data manipulation task or query. Stop adding in layers of Airflow/Dagster/Prefect when a simple cron would work. Stop adding in Kubernetes/GKE/Fargate to manage the aforementioned. Stop moving data between systems constantly (meltano, airbyte, fivetran) when you already have it in a perfectly good place. Stop with the toxic positivity that's completely overflowing the modern-data world and all these bullshit VC-infused startups who are convinced they need every single element and more. 99% of business needs can be satisfied by a single Postgres/Mysql installation and a halfway-competent person armed with SQL and an understanding of normalization. Reach for Excel when you need to do more "hands-on" analysis, business modeling, charting, basic forecasting, and presentation for non-technical users. [1] https://mode.com/sql-tutorial/ https://mode.com/sql-tutorial/ [2] https://use-the-index-luke.com/ https://use-the-index-luke.com/
- b-luu 4y ago
  Good points, but a bit extremist. I definitely agree with the over-hype making simple mundane tasks way harder than they should be. So yeah, do NOT over-engineer!! But, on the other hand, doing everything with a single Postgres and spreadsheets seems to go with the hammer-to-nail adage. And all too often, you end up with unmaintainable duck-taped hack-arounds... Which is clearly NOT better (nor necessarily worst) than the over-engineered solution. In some cases (maybe not 1% but clearly not the majority either), it does make sense to look at other tools that might be available. That being said, there are waaaay too many options to filter through, because of that darn hype bubble.

by b-luu 4y ago

datalopers 4y ago

> how to make the modern data stack more intuitive? I'd start with getting people to learn relational data modeling and SQL [1][2] at a deep level. Stop reaching for python and pandas/spark for every basic data manipulation task or query. Stop adding in layers of Airflow/Dagster/Prefect when a simple cron would work. Stop adding in Kubernetes/GKE/Fargate to manage the aforementioned. Stop moving data between systems constantly (meltano, airbyte, fivetran) when you already have it in a perfectly good place. Stop with the toxic positivity that's completely overflowing the modern-data world and all these bullshit VC-infused startups who are convinced they need every single element and more. 99% of business needs can be satisfied by a single Postgres/Mysql installation and a halfway-competent person armed with SQL and an understanding of normalization. Reach for Excel when you need to do more "hands-on" analysis, business modeling, charting, basic forecasting, and presentation for non-technical users. [1] https://mode.com/sql-tutorial/ https://mode.com/sql-tutorial/ [2] https://use-the-index-luke.com/ https://use-the-index-luke.com/

b-luu 4y ago
Good points, but a bit extremist. I definitely agree with the over-hype making simple mundane tasks way harder than they should be. So yeah, do NOT over-engineer!! But, on the other hand, doing everything with a single Postgres and spreadsheets seems to go with the hammer-to-nail adage. And all too often, you end up with unmaintainable duck-taped hack-arounds... Which is clearly NOT better (nor necessarily worst) than the over-engineered solution. In some cases (maybe not 1% but clearly not the majority either), it does make sense to look at other tools that might be available. That being said, there are waaaay too many options to filter through, because of that darn hype bubble.