4 ms·

Yes, but such situations are incredibly rare IMO. I was just recently in a job where a self described expert was obsessed with writing all data pipelines (in th

by extr 2y ago

Yes, but such situations are incredibly rare IMO. I was just recently in a job where a self described expert was obsessed with writing all data pipelines (in this context, 1-2GB of data) in spark, because that was the “proper” way to do things since we had Databricks. It was beyond a waste of time.

tmountain 2y ago
Yup, everyone thought they had “big data” for a moment in the industry, and it turns out, they all just read the same trendy blog posts. I made the mistake of “clustering” at my previous company, and it introduced so much unnecessary complexity. If you are really worried about future proofing, just design a schema where the main entities are denormalized and use UUID for your primary keys. If you need to shard later, it will be much easier. Although, again, it’s likely that you will never need to.
mr_toad 2y ago
> Yes, but such situations are incredibly rare IMO. They happen all the time if you work for banks, large finance companies or government. It’s not just the 2TB databases - it’s the 100 analysts all doing it their own thing with that data at the same time.
- extr 2y ago
  Maybe "rare" was the wrong choice of word. Maybe I should have said such situations are "highly contextual". It's one of those things where if you have to ask the question "Do I need this?" you almost certainly don't. And even in situations where you actually do need it, it's strongly preferred to "find a way not to need it" for day to day things (aka downsample, or be judicious about what you load into the analytics environment). I can only speak to my experience in FAANG, but even though we had the infra and budget to run huge distributed queries, it was highly discouraged unless absolutely needed! This is my problem with databricks. It seems like in the course of selling their product they have taken the received wisdom of "do not run expensive and complex compute clusters unless absolutely necessary" and turned it into "it's fun and easy to run distributed compute clusters - everyone's doing it and you should too" regardless of how contextually appropriate it is.

by extr 2y ago

tmountain 2y ago

Yup, everyone thought they had “big data” for a moment in the industry, and it turns out, they all just read the same trendy blog posts. I made the mistake of “clustering” at my previous company, and it introduced so much unnecessary complexity. If you are really worried about future proofing, just design a schema where the main entities are denormalized and use UUID for your primary keys. If you need to shard later, it will be much easier. Although, again, it’s likely that you will never need to.

mr_toad 2y ago

> Yes, but such situations are incredibly rare IMO. They happen all the time if you work for banks, large finance companies or government. It’s not just the 2TB databases - it’s the 100 analysts all doing it their own thing with that data at the same time.

extr 2y ago
Maybe "rare" was the wrong choice of word. Maybe I should have said such situations are "highly contextual". It's one of those things where if you have to ask the question "Do I need this?" you almost certainly don't. And even in situations where you actually do need it, it's strongly preferred to "find a way not to need it" for day to day things (aka downsample, or be judicious about what you load into the analytics environment). I can only speak to my experience in FAANG, but even though we had the infra and budget to run huge distributed queries, it was highly discouraged unless absolutely needed! This is my problem with databricks. It seems like in the course of selling their product they have taken the received wisdom of "do not run expensive and complex compute clusters unless absolutely necessary" and turned it into "it's fun and easy to run distributed compute clusters - everyone's doing it and you should too" regardless of how contextually appropriate it is.