3 ms·
Sharding = partitioning and it's the only way data warehouses can scale to handle large datasets. Tables are partitioned on the primary key, usually with range-
by manigandham 5y ago
Sharding = partitioning and it's the only way data warehouses can scale to handle large datasets. Tables are partitioned on the primary key, usually with range-based sharding for flexibility.
Some warehouses like BigQuery force this as a numeric key (dates by default) while others leave it up to you. Partitioning can also use multiple columns to help with more even distribution and/or better query performance.
- tremon 5y agoThe technical difference between sharding and partitioning is at which level the split lies. If you're splitting the data storage within a single database, it's called partitioning. If you're splitting the data across multiple databases instead, it's sharding. But you can't implement either if you haven't logically partitioned your data first.
- i_like_waiting 5y agoyeah, partitioning is very good at almost any level, with very few exceptions. Sharding on other hand is pure tradeoff, I haven't looked into hard, but I guess 1 computer, where database sits, cannot have unlimited RAIDs of hard drives. So question is, should I push for 1 server only as long as possible, or when its better to shard between multiple?