4 ms·

Ahh yes, sorry, you are running it on a single instance. That’s capped, but the aggregate throughput across instances can be a lot higher. There are request li

by orf 4y ago

Ahh yes, sorry, you are running it on a single instance. That’s capped, but the aggregate throughput across instances can be a lot higher. There are request limits but these are per partition. There is also an undocumented hard cap on list requests per second which is much lower than get object.

zX41ZdbW 4y ago
No, I'm creating 10, 100, and 500 different EC2 instances and measuring the aggregate throughput.
- zX41ZdbW 4y ago
  100 instances of m5.8xlarge, 500 instances of m5.2xlarge
  - orf 4y ago
    Something must be wrong, at my previous work we were able to achieve a much higher aggregate throughput - are you spreading them across AZ, and are you using a VPC endpoint? We did use multiple VPCs, that might make a difference
    - zX41ZdbW 4y ago
      Yes, the current experiment was run from a single VPC. It should explain the difference.
      - orf 4y ago
        Why not randomise VPCs between instances? It’s fairly easy to do with a fixed pool and random selection.
- zX41ZdbW 4y ago
  Worth to note that ClickHouse is a distributed MPP DBMS, it can scale up to 1000s of bare-metal servers and process terabytes (not terabits) of data per second. It also works on a laptop, or even without installation.