3 ms·
> how to make the modern data stack more intuitive? I'd start with getting people to learn relational data modeling and SQL [1][2] at a deep level. Stop reachi
by datalopers 5y ago
> how to make the modern data stack more intuitive?
I'd start with getting people to learn relational data modeling and SQL [1][2] at a deep level. Stop reaching for python and pandas/spark for every basic data manipulation task or query. Stop adding in layers of Airflow/Dagster/Prefect when a simple cron would work. Stop adding in Kubernetes/GKE/Fargate to manage the aforementioned. Stop moving data between systems constantly (meltano, airbyte, fivetran) when you already have it in a perfectly good place. Stop with the toxic positivity that's completely overflowing the modern-data world and all these bullshit VC-infused startups who are convinced they need every single element and more.
99% of business needs can be satisfied by a single Postgres/Mysql installation and a halfway-competent person armed with SQL and an understanding of normalization. Reach for Excel when you need to do more "hands-on" analysis, business modeling, charting, basic forecasting, and presentation for non-technical users.
[1] https://mode.com/sql-tutorial/ https://mode.com/sql-tutorial/
[2] https://use-the-index-luke.com/ https://use-the-index-luke.com/
- b-luu 5y agoGood points, but a bit extremist. I definitely agree with the over-hype making simple mundane tasks way harder than they should be. So yeah, do NOT over-engineer!! But, on the other hand, doing everything with a single Postgres and spreadsheets seems to go with the hammer-to-nail adage. And all too often, you end up with unmaintainable duck-taped hack-arounds... Which is clearly NOT better (nor necessarily worst) than the over-engineered solution. In some cases (maybe not 1% but clearly not the majority either), it does make sense to look at other tools that might be available. That being said, there are waaaay too many options to filter through, because of that darn hype bubble.