4 ms·

See my comment above. Aurora partitions ('shards') its data in 10GB chunks automatically, which are written in parallel to 6 nodes across 3 Availability Zones a

See my comment above. Aurora partitions ('shards') its data in 10GB chunks automatically, which are written in parallel to 6 nodes across 3 Availability Zones and backed up to S3 for full durability. Aurora horizontally scales storage capacity and read throughput (and with multi-master currently in preview, write throughput) without requiring the application to handle any implementation details of 'sharding'. This is a feature, not a bug.

ddorian43 9y ago
See my comment again. The data needs to be partitioned (looks like it is in 10 GB chunks), can 6 storage servers go 600 ? And looks like the compute layer can't pushdown processing to storage servers. You do a "select coun(*) from t where a=1" and "a is not indexed" on 600 storage servers and 1 compute server. Does the count happen in parallel on 600 storage-nodes and 1 accumulate on the compute server ? Or rows transfer to compute for counting ?
- wgjordan 9y ago
  > can 6 storage servers go 600? Yes, 6 storage servers (10GB chunk) automatically scale up to 6000+ (64TB) [1]. > And looks like the compute layer can't pushdown processing to storage servers. 'Parallel query' for Aurora became available for preview in February [2]. The compute layer does pushdown processing to storage servers [3]. If you do a "select count(*) from t where a=1" and "a is not indexed" on 600 storage servers and 1 compute server, the count should happen in parallel on 600 storage nodes and 1 accumulate on the compute server. [1]: https://aws.amazon.com/rds/aurora/faqs/#hardware-and-scaling https://aws.amazon.com/rds/aurora/faqs/#hardware-and-scaling [2]: https://aws.amazon.com/about-aws/whats-new/2018/02/amazon-aurora-parallel-query-is-available-for-preview/ https://aws.amazon.com/about-aws/whats-new/2018/02/amazon-au... [3]: https://www.slideshare.net/AmazonWebServices/deep-dive-on-the-amazon-aurora-mysqlcompatible-edition-dat301-reinvent-2017/41 https://www.slideshare.net/AmazonWebServices/deep-dive-on-th...
  - ddorian43 9y ago
    ok I'm sorry, I wasn't aware, you're right
    - elvinyung 9y ago
      To add to what GP said, I'm pretty sure the current storage limitation is a control plane-related bottleneck, not because of "vertical scaling" in storage.

ddorian43 9y ago

See my comment again. The data needs to be partitioned (looks like it is in 10 GB chunks), can 6 storage servers go 600 ? And looks like the compute layer can't pushdown processing to storage servers. You do a "select coun(*) from t where a=1" and "a is not indexed" on 600 storage servers and 1 compute server. Does the count happen in parallel on 600 storage-nodes and 1 accumulate on the compute server ? Or rows transfer to compute for counting ?

wgjordan 9y ago
> can 6 storage servers go 600? Yes, 6 storage servers (10GB chunk) automatically scale up to 6000+ (64TB) [1]. > And looks like the compute layer can't pushdown processing to storage servers. 'Parallel query' for Aurora became available for preview in February [2]. The compute layer does pushdown processing to storage servers [3]. If you do a "select count(*) from t where a=1" and "a is not indexed" on 600 storage servers and 1 compute server, the count should happen in parallel on 600 storage nodes and 1 accumulate on the compute server. [1]: https://aws.amazon.com/rds/aurora/faqs/#hardware-and-scaling https://aws.amazon.com/rds/aurora/faqs/#hardware-and-scaling [2]: https://aws.amazon.com/about-aws/whats-new/2018/02/amazon-aurora-parallel-query-is-available-for-preview/ https://aws.amazon.com/about-aws/whats-new/2018/02/amazon-au... [3]: https://www.slideshare.net/AmazonWebServices/deep-dive-on-the-amazon-aurora-mysqlcompatible-edition-dat301-reinvent-2017/41 https://www.slideshare.net/AmazonWebServices/deep-dive-on-th...
- ddorian43 9y ago
  ok I'm sorry, I wasn't aware, you're right
  - elvinyung 9y ago
    To add to what GP said, I'm pretty sure the current storage limitation is a control plane-related bottleneck, not because of "vertical scaling" in storage.