3 ms·
Does sharding work for splitting DWH tables? I guess it has to, but whats strategy there, if its mostly about "slicing and dicing" in every way possible. Mayb
by i_like_waiting 5y ago
Does sharding work for splitting DWH tables?
I guess it has to, but whats strategy there, if its mostly about "slicing and dicing" in every way possible.
Maybe recent vs. historical data, or some known customer segments, extracted from previous analysis?
- snissn 5y agoDate partitions very good. Also other enums or even client ids can be great partitions (in addition simultaneously with date)
- manigandham 5y agoSharding = partitioning and it's the only way data warehouses can scale to handle large datasets. Tables are partitioned on the primary key, usually with range-based sharding for flexibility. Some warehouses like BigQuery force this as a numeric key (dates by default) while others leave it up to you. Partitioning can also use multiple columns to help with more even distribution and/or better query performance.
- tremon 5y agoThe technical difference between sharding and partitioning is at which level the split lies. If you're splitting the data storage within a single database, it's called partitioning. If you're splitting the data across multiple databases instead, it's sharding. But you can't implement either if you haven't logically partitioned your data first.
- i_like_waiting 5y agoyeah, partitioning is very good at almost any level, with very few exceptions. Sharding on other hand is pure tradeoff, I haven't looked into hard, but I guess 1 computer, where database sits, cannot have unlimited RAIDs of hard drives. So question is, should I push for 1 server only as long as possible, or when its better to shard between multiple?
- thejosh 5y agoHow big your data is also dictates your strategy. And also your software. What works for say snowflake night not work for others