5 ms·
See my comment again. The data needs to be partitioned (looks like it is in 10 GB chunks), can 6 storage servers go 600 ? And looks like the compute layer can'
by ddorian43 9y ago
See my comment again. The data needs to be partitioned (looks like it is in 10 GB chunks), can 6 storage servers go 600 ?
And looks like the compute layer can't pushdown processing to storage servers.
You do a "select coun(*) from t where a=1" and "a is not indexed" on 600 storage servers and 1 compute server. Does the count happen in parallel on 600 storage-nodes and 1 accumulate on the compute server ? Or rows transfer to compute for counting ?
- wgjordan 9y ago> can 6 storage servers go 600? Yes, 6 storage servers (10GB chunk) automatically scale up to 6000+ (64TB) [1]. > And looks like the compute layer can't pushdown processing to storage servers. 'Parallel query' for Aurora became available for preview in February [2]. The compute layer does pushdown processing to storage servers [3]. If you do a "select count(*) from t where a=1" and "a is not indexed" on 600 storage servers and 1 compute server, the count should happen in parallel on 600 storage nodes and 1 accumulate on the compute server. [1]: https://aws.amazon.com/rds/aurora/faqs/#hardware-and-scaling https://aws.amazon.com/rds/aurora/faqs/#hardware-and-scaling [2]: https://aws.amazon.com/about-aws/whats-new/2018/02/amazon-aurora-parallel-query-is-available-for-preview/ https://aws.amazon.com/about-aws/whats-new/2018/02/amazon-au... [3]: https://www.slideshare.net/AmazonWebServices/deep-dive-on-the-amazon-aurora-mysqlcompatible-edition-dat301-reinvent-2017/41 https://www.slideshare.net/AmazonWebServices/deep-dive-on-th...