3 ms·

Even with a large database servers (10s of CPU cores, 100s of gigabytes of RAM) bottlenecks arise pretty quickly. Err, do they? For what percent of real world

by themgt 3mo ago

Even with a large database servers (10s of CPU cores, 100s of gigabytes of RAM) bottlenecks arise pretty quickly. Err, do they? For what percent of real world use cases? The database can scale to handle more traffic by adding replicas. An extreme example of this is OpenAI's use of 50 replicas on a single Primary. So an extreme example is OpenAI needing 50 replicas, but we're doing five blades ... err, we're doing 768 servers because the need arose "pretty quickly"? When we needed to store a petabyte of data (one million gigabytes), we'd need many more shards For who? The United States government? How many end-users are running 1PB Postgres database on DBaaS?

exiguus 3mo ago
Personally, I know several databases where single tables have +500GB and the database has +100TB. With this huge databases the restore and backup process over network become indeed a bottleneck. So I can agree with the author. Also, the author does not say that they can't start with a single database server and just read replicas and max hardware out. Real world use cases with +100TB I know about are Stock Market, Traffic Data, Warehouses, Analytics and Monitoring.
bddicken 3mo ago
> So an extreme example is OpenAI needing 50 replicas, but we're doing five blades ... err, we're doing 768 servers because the need arose "pretty quickly"? If you read the OpenAI article, you'll see that they actually used sharding to offload a bunch of work from their "1 primary 50 replicas setup" >>> "To mitigate these limitations and reduce write pressure, we’ve migrated, and continue to migrate, shardable (i.e. workloads that can be horizontally partitioned), write-heavy workloads to sharded systems such as Azure Cosmos DB..." The 768 servers and 1PB example is just one of many configurations. A business with 10TB may choose to go from a monolithic database to a 8-shard setup to improve backup times, eliminate single-point-of-failure, have more breathing room for scaling.

by themgt 3mo ago

exiguus 3mo ago

Personally, I know several databases where single tables have +500GB and the database has +100TB. With this huge databases the restore and backup process over network become indeed a bottleneck. So I can agree with the author. Also, the author does not say that they can't start with a single database server and just read replicas and max hardware out. Real world use cases with +100TB I know about are Stock Market, Traffic Data, Warehouses, Analytics and Monitoring.

bddicken 3mo ago

> So an extreme example is OpenAI needing 50 replicas, but we're doing five blades ... err, we're doing 768 servers because the need arose "pretty quickly"? If you read the OpenAI article, you'll see that they actually used sharding to offload a bunch of work from their "1 primary 50 replicas setup" >>> "To mitigate these limitations and reduce write pressure, we’ve migrated, and continue to migrate, shardable (i.e. workloads that can be horizontally partitioned), write-heavy workloads to sharded systems such as Azure Cosmos DB..." The 768 servers and 1PB example is just one of many configurations. A business with 10TB may choose to go from a monolithic database to a 8-shard setup to improve backup times, eliminate single-point-of-failure, have more breathing room for scaling.