3 ms·

You won’t be throttled if you partition your data “correctly” and have a reasonable amount of data in your bucket. That’s difficult to do, but more than possibl

by orf 4y ago

You won’t be throttled if you partition your data “correctly” and have a reasonable amount of data in your bucket. That’s difficult to do, but more than possible.

zX41ZdbW 4y ago
We already partition our data correctly, as the AWS solution architects recommend, but there are limitations on the: - total throughput (100 Gbit/sec); - the number of requests. For example, at this moment, I'm doing an experiment: creating 10, 100, and 500 Clickhouse servers, and reading the data from s3, either from a MergeTree table or from a set of files. Ten instances saturate 100 Gbit/sec bandwidth, and there is no subsequent improvement. JFYI, 100 Gbit/sec is less than one PCI-e with a few M.2 SSDs can give.
- zX41ZdbW 4y ago
  https://repost.aws/knowledge-center/s3-maximum-transfer-speed-ec2 https://repost.aws/knowledge-center/s3-maximum-transfer-spee...
- orf 4y ago
  Ahh yes, sorry, you are running it on a single instance. That’s capped, but the aggregate throughput across instances can be a lot higher. There are request limits but these are per partition. There is also an undocumented hard cap on list requests per second which is much lower than get object.
  - zX41ZdbW 4y ago
    No, I'm creating 10, 100, and 500 different EC2 instances and measuring the aggregate throughput.
    - zX41ZdbW 4y ago
      100 instances of m5.8xlarge, 500 instances of m5.2xlarge
      - orf 4y ago
        Something must be wrong, at my previous work we were able to achieve a much higher aggregate throughput - are you spreading them across AZ, and are you using a VPC endpoint? We did use multiple VPCs, that might make a difference
        zX41ZdbW 4y ago
        Yes, the current experiment was run from a single VPC. It should explain the difference.
        orf 4y ago
        Why not randomise VPCs between instances? It’s fairly easy to do with a fixed pool and random selection.
    - zX41ZdbW 4y ago
      Worth to note that ClickHouse is a distributed MPP DBMS, it can scale up to 1000s of bare-metal servers and process terabytes (not terabits) of data per second. It also works on a laptop, or even without installation.
zX41ZdbW 4y ago
https://imgur.com/a/uh8H9PO https://imgur.com/a/uh8H9PO

by orf 4y ago

You won’t be throttled if you partition your data “correctly” and have a reasonable amount of data in your bucket. That’s difficult to do, but more than possible.

zX41ZdbW 4y ago

We already partition our data correctly, as the AWS solution architects recommend, but there are limitations on the: - total throughput (100 Gbit/sec); - the number of requests. For example, at this moment, I'm doing an experiment: creating 10, 100, and 500 Clickhouse servers, and reading the data from s3, either from a MergeTree table or from a set of files. Ten instances saturate 100 Gbit/sec bandwidth, and there is no subsequent improvement. JFYI, 100 Gbit/sec is less than one PCI-e with a few M.2 SSDs can give.

zX41ZdbW 4y ago
https://repost.aws/knowledge-center/s3-maximum-transfer-speed-ec2 https://repost.aws/knowledge-center/s3-maximum-transfer-spee...
orf 4y ago
Ahh yes, sorry, you are running it on a single instance. That’s capped, but the aggregate throughput across instances can be a lot higher. There are request limits but these are per partition. There is also an undocumented hard cap on list requests per second which is much lower than get object.
- zX41ZdbW 4y ago
  No, I'm creating 10, 100, and 500 different EC2 instances and measuring the aggregate throughput.
  - zX41ZdbW 4y ago
    100 instances of m5.8xlarge, 500 instances of m5.2xlarge
    - orf 4y ago
      Something must be wrong, at my previous work we were able to achieve a much higher aggregate throughput - are you spreading them across AZ, and are you using a VPC endpoint? We did use multiple VPCs, that might make a difference
      - zX41ZdbW 4y ago
        Yes, the current experiment was run from a single VPC. It should explain the difference.
        orf 4y ago
        Why not randomise VPCs between instances? It’s fairly easy to do with a fixed pool and random selection.
  - zX41ZdbW 4y ago
    Worth to note that ClickHouse is a distributed MPP DBMS, it can scale up to 1000s of bare-metal servers and process terabytes (not terabits) of data per second. It also works on a laptop, or even without installation.

zX41ZdbW 4y ago

https://imgur.com/a/uh8H9PO https://imgur.com/a/uh8H9PO