26 ms·

Our experiments all ran on cloud instances (DO and Timescale Forge) that use storage that are replicated across multiple AZs/racks for greater reliability and f

Our experiments all ran on cloud instances (DO and Timescale Forge) that use storage that are replicated across multiple AZs/racks for greater reliability and fault tolerance.

rubiquity 6y ago
Nice! Multiple AZs or Multiple racks? EDIT: Also your response contradicts what is in the blog post: > 1 remote client machine, 1 database server, both in the same cloud datacenter > Disk Size: 4.8TB of disk in a raid0 configuration (EXT4 filesystem)
- mfreed 6y ago
  On our managed Timescale Forge, it's multiple AZ. (On some benchmarking equipment on Digital Ocean, it's advertised as "multiple racks", but managed to reduce blast radius.)
  - rubiquity 6y ago
    This contradicts what is in the blog post. From the machine configuration section: > 1 remote client machine, 1 database server, both in the same cloud datacenter > Disk Size: 4.8TB of disk in a raid0 configuration (EXT4 filesystem) Both those statements lead me to believe it's a single server with locally attached SSDs in a RAID0. Which is it? I know benchmarking is hard, and it's difficult to test certain aspects of Amazon Timestream due to it being a managed service, but I really think these details need to be firmed up to make sure you are comparing apples to apples. TimescaleDB seems like a cool product. Another suggestion I'd make is to run the Amazon Timestream clients across multiple AZs if you aren't already. The blog post doesn't mention whether all the t3 instances are in the same AZ or not.
    - ryanbooz 6y ago
      > Another suggestion I'd make is to run the Amazon Timestream clients across multiple AZs if you aren't already. The blog post doesn't mention whether all the t3 instances are in the same AZ or not. They were all run in the same AZ. This brings up a good point, however, that we discussed internally when the first results came back. If there are tricks like this that might improve performance, it's not (currently) spelled out in the documentation so there's no way to know that. And we reached out for help in various forums with no response. It's worth noting that since we performed this analysis, Amazon did release their own tooling for a similar benchmark and created a post[0]. While neither it, nor the tooling documentation[1] specifically spell out how many threads or instances they ran to achieve their results, it's hard to draw an apples-to-apples comparison. It does reveal that they used (had to use??) an m5.24xlarge instance (96 vCPU,384GB) to run their tests. As discussed in the article, one much smaller t3 instance was able to add >1 million metrics/second into TimescaleDB running in Timescale Forge. [0] https://aws.amazon.com/blogs/database/deriving-real-time-insights-over-petabytes-of-time-series-data-with-amazon-timestream/ https://aws.amazon.com/blogs/database/deriving-real-time-ins... [1] https://github.com/awslabs/amazon-timestream-tools/tree/master/tools/perf-scale-workload https://github.com/awslabs/amazon-timestream-tools/tree/mast...
    - mfreed 6y ago
      It's raided network-attached block storage. We'll update the blog post to make that clear.

Our experiments all ran on cloud instances (DO and Timescale Forge) that use storage that are replicated across multiple AZs/racks for greater reliability and fault tolerance.

rubiquity 6y ago

Nice! Multiple AZs or Multiple racks? EDIT: Also your response contradicts what is in the blog post: > 1 remote client machine, 1 database server, both in the same cloud datacenter > Disk Size: 4.8TB of disk in a raid0 configuration (EXT4 filesystem)

mfreed 6y ago
On our managed Timescale Forge, it's multiple AZ. (On some benchmarking equipment on Digital Ocean, it's advertised as "multiple racks", but managed to reduce blast radius.)
- rubiquity 6y ago
  This contradicts what is in the blog post. From the machine configuration section: > 1 remote client machine, 1 database server, both in the same cloud datacenter > Disk Size: 4.8TB of disk in a raid0 configuration (EXT4 filesystem) Both those statements lead me to believe it's a single server with locally attached SSDs in a RAID0. Which is it? I know benchmarking is hard, and it's difficult to test certain aspects of Amazon Timestream due to it being a managed service, but I really think these details need to be firmed up to make sure you are comparing apples to apples. TimescaleDB seems like a cool product. Another suggestion I'd make is to run the Amazon Timestream clients across multiple AZs if you aren't already. The blog post doesn't mention whether all the t3 instances are in the same AZ or not.
  - ryanbooz 6y ago
    > Another suggestion I'd make is to run the Amazon Timestream clients across multiple AZs if you aren't already. The blog post doesn't mention whether all the t3 instances are in the same AZ or not. They were all run in the same AZ. This brings up a good point, however, that we discussed internally when the first results came back. If there are tricks like this that might improve performance, it's not (currently) spelled out in the documentation so there's no way to know that. And we reached out for help in various forums with no response. It's worth noting that since we performed this analysis, Amazon did release their own tooling for a similar benchmark and created a post[0]. While neither it, nor the tooling documentation[1] specifically spell out how many threads or instances they ran to achieve their results, it's hard to draw an apples-to-apples comparison. It does reveal that they used (had to use??) an m5.24xlarge instance (96 vCPU,384GB) to run their tests. As discussed in the article, one much smaller t3 instance was able to add >1 million metrics/second into TimescaleDB running in Timescale Forge. [0] https://aws.amazon.com/blogs/database/deriving-real-time-insights-over-petabytes-of-time-series-data-with-amazon-timestream/ https://aws.amazon.com/blogs/database/deriving-real-time-ins... [1] https://github.com/awslabs/amazon-timestream-tools/tree/master/tools/perf-scale-workload https://github.com/awslabs/amazon-timestream-tools/tree/mast...
  - mfreed 6y ago
    It's raided network-attached block storage. We'll update the blog post to make that clear.