5 ms·

Databricks/Spark is overpriced garbage for Python-toting data scientists/engineers/analysts who don't know how to model data or write SQL. Exactly like Hadoop w

Databricks/Spark is overpriced garbage for Python-toting data scientists/engineers/analysts who don't know how to model data or write SQL. Exactly like Hadoop was 15 years ago for the Java-toting data scientists/engineers.

clpm4j 4y ago
That might be hyperbolic, but directionally I agree. The interesting part is that there are enough of those people out there in enterprise who fall into that "don't really know what they're doing but want to try something cool" bucket to keep Databricks flush with cash for years on end.
- datalopers 4y ago
  Yep. MongoDB earned $873M in revenue over the past 12 months servicing tech debt created between ‘10 - ‘16. Databricks will be around a long time for the same reason. It’s a viable business strategy for sure.
- michael_j_ward 4y ago
  If one does not need petabytes of scale, but am otherwise interested in the data lineage / observability / workflow being sold by databricks, what would you suggest? Some evaluation Criteria: - Ease of maintenance and operation is almost paramount. - It's fine if the solution never lives anywhere but 1 single virtual server that scales vertically (data might grow to a couple TB, but not PETA BYTES) - Similarly, 20 9's is not a criteria. If the machine fails and it takes an hour till someone goes and re-deploys, that's fine. - Declarative, reproducible deployment with an easy upgrade story would be great - Ideal if the deployment can be run locally for quick developmnet
  - nooorofe 4y ago
    Talend Studio? https://www.talend.com/products/talend-open-studio/ https://www.talend.com/products/talend-open-studio/
alexott 4y ago
Databricks includes pure SQL implementations - Databricks SQL, for example. Or for example, Delta Live Tables allow to write you pipelines using only SQL…
threeseed 4y ago
Spark is open source and free. Databricks is their hosted version just like Aurora is a hosted PostgreSQL. The fact you don't seem to understand the difference makes me question if you are qualified to be making statements that it is garbage and we can simply write SQL. Because having worked on many large, big data projects SQL is often not the right tool for every use case.
- deleted 4y ago
  [deleted]
nojito 4y ago
Spark is many orders of magnitude faster than "properly modeled" data
- deleted 4y ago
  [deleted]
lysecret 4y ago
As a former "Python-toting data scientists/engineers/analysts who don't know how to model data or write SQL" I wholeheartedly agree.

clpm4j 4y ago

That might be hyperbolic, but directionally I agree. The interesting part is that there are enough of those people out there in enterprise who fall into that "don't really know what they're doing but want to try something cool" bucket to keep Databricks flush with cash for years on end.

datalopers 4y ago
Yep. MongoDB earned $873M in revenue over the past 12 months servicing tech debt created between ‘10 - ‘16. Databricks will be around a long time for the same reason. It’s a viable business strategy for sure.
michael_j_ward 4y ago
If one does not need petabytes of scale, but am otherwise interested in the data lineage / observability / workflow being sold by databricks, what would you suggest? Some evaluation Criteria: - Ease of maintenance and operation is almost paramount. - It's fine if the solution never lives anywhere but 1 single virtual server that scales vertically (data might grow to a couple TB, but not PETA BYTES) - Similarly, 20 9's is not a criteria. If the machine fails and it takes an hour till someone goes and re-deploys, that's fine. - Declarative, reproducible deployment with an easy upgrade story would be great - Ideal if the deployment can be run locally for quick developmnet
- nooorofe 4y ago
  Talend Studio? https://www.talend.com/products/talend-open-studio/ https://www.talend.com/products/talend-open-studio/

alexott 4y ago

Databricks includes pure SQL implementations - Databricks SQL, for example. Or for example, Delta Live Tables allow to write you pipelines using only SQL…

threeseed 4y ago

Spark is open source and free. Databricks is their hosted version just like Aurora is a hosted PostgreSQL. The fact you don't seem to understand the difference makes me question if you are qualified to be making statements that it is garbage and we can simply write SQL. Because having worked on many large, big data projects SQL is often not the right tool for every use case.

deleted 4y ago
[deleted]

nojito 4y ago

Spark is many orders of magnitude faster than "properly modeled" data

deleted 4y ago
[deleted]

lysecret 4y ago

As a former "Python-toting data scientists/engineers/analysts who don't know how to model data or write SQL" I wholeheartedly agree.