4 ms·

Disclosure: I work at AWS but not on Timestream. Opinions my own. Unless I'm missing something this is not an apples to apples benchmark. TimescaleDB is runnin

Disclosure: I work at AWS but not on Timestream. Opinions my own. Unless I'm missing something this is not an apples to apples benchmark. TimescaleDB is running as a single node without any replication whereas Amazon Timestream is replicated[0] to three AWS Availability Zones for durability. I've only skimmed the TSBS[1] repo and the start/stop scripts for TimescaleDB. Can someone confirm this? 0 - https://aws.amazon.com/blogs/aws/store-and-access-time-series-data-at-any-scale-with-amazon-timestream-now-generally-available/ https://aws.amazon.com/blogs/aws/store-and-access-time-serie... 1 - https://github.com/timescale/tsbs https://github.com/timescale/tsbs

ryanbooz 6y ago
Our experiments all ran on cloud instances (DO and Timescale Forge) that use storage that are replicated across multiple AZs/racks for greater reliability and fault tolerance.
- rubiquity 6y ago
  Nice! Multiple AZs or Multiple racks? EDIT: Also your response contradicts what is in the blog post: > 1 remote client machine, 1 database server, both in the same cloud datacenter > Disk Size: 4.8TB of disk in a raid0 configuration (EXT4 filesystem)
  - mfreed 6y ago
    On our managed Timescale Forge, it's multiple AZ. (On some benchmarking equipment on Digital Ocean, it's advertised as "multiple racks", but managed to reduce blast radius.)
    - rubiquity 6y ago
      This contradicts what is in the blog post. From the machine configuration section: > 1 remote client machine, 1 database server, both in the same cloud datacenter > Disk Size: 4.8TB of disk in a raid0 configuration (EXT4 filesystem) Both those statements lead me to believe it's a single server with locally attached SSDs in a RAID0. Which is it? I know benchmarking is hard, and it's difficult to test certain aspects of Amazon Timestream due to it being a managed service, but I really think these details need to be firmed up to make sure you are comparing apples to apples. TimescaleDB seems like a cool product. Another suggestion I'd make is to run the Amazon Timestream clients across multiple AZs if you aren't already. The blog post doesn't mention whether all the t3 instances are in the same AZ or not.
      - ryanbooz 6y ago
        > Another suggestion I'd make is to run the Amazon Timestream clients across multiple AZs if you aren't already. The blog post doesn't mention whether all the t3 instances are in the same AZ or not. They were all run in the same AZ. This brings up a good point, however, that we discussed internally when the first results came back. If there are tricks like this that might improve performance, it's not (currently) spelled out in the documentation so there's no way to know that. And we reached out for help in various forums with no response. It's worth noting that since we performed this analysis, Amazon did release their own tooling for a similar benchmark and created a post[0]. While neither it, nor the tooling documentation[1] specifically spell out how many threads or instances they ran to achieve their results, it's hard to draw an apples-to-apples comparison. It does reveal that they used (had to use??) an m5.24xlarge instance (96 vCPU,384GB) to run their tests. As discussed in the article, one much smaller t3 instance was able to add >1 million metrics/second into TimescaleDB running in Timescale Forge. [0] https://aws.amazon.com/blogs/database/deriving-real-time-insights-over-petabytes-of-time-series-data-with-amazon-timestream/ https://aws.amazon.com/blogs/database/deriving-real-time-ins... [1] https://github.com/awslabs/amazon-timestream-tools/tree/master/tools/perf-scale-workload https://github.com/awslabs/amazon-timestream-tools/tree/mast...
      - mfreed 6y ago
        It's raided network-attached block storage. We'll update the blog post to make that clear.
overcast 6y ago
Would that really explain the difference between 5 minutes on Timescale and 2 weeks on Timestream? Timeseries databases need the ability to ingest data rapidly.
- ryanbooz 6y ago
  Please see below for correction... but at some point today a poster said "~2 weeks" for Timestream... but that's incorrect. We ran the data load/ingest for "~2 days" (40 hours). Just want to make sure the right numbers are being used! Thanks! (Post author @Timescale)
  - overcast 6y ago
    STILL! 5 minutes vs 2 DAYS can't be attributed to just being replicated.
    - rubiquity 6y ago
      Large numbers are unintuitive. If we’re conservative and add only 1 millisecond of latency for the 3 AZ replication, on 410 million records that is about 4 and a half days of time if the client is not doing any batching.
      - ryanbooz 6y ago
        TSBS did batch 100 metrics at a time, the max per request that Timestream allows. As we (and others) have suggested, this could easily be one of the reasons it's harder to ingest more quickly without significantly more threads. TimescaleDB (and other TSDB's) allow much higher batching values.

ryanbooz 6y ago

Our experiments all ran on cloud instances (DO and Timescale Forge) that use storage that are replicated across multiple AZs/racks for greater reliability and fault tolerance.

rubiquity 6y ago
Nice! Multiple AZs or Multiple racks? EDIT: Also your response contradicts what is in the blog post: > 1 remote client machine, 1 database server, both in the same cloud datacenter > Disk Size: 4.8TB of disk in a raid0 configuration (EXT4 filesystem)
- mfreed 6y ago
  On our managed Timescale Forge, it's multiple AZ. (On some benchmarking equipment on Digital Ocean, it's advertised as "multiple racks", but managed to reduce blast radius.)
  - rubiquity 6y ago
    This contradicts what is in the blog post. From the machine configuration section: > 1 remote client machine, 1 database server, both in the same cloud datacenter > Disk Size: 4.8TB of disk in a raid0 configuration (EXT4 filesystem) Both those statements lead me to believe it's a single server with locally attached SSDs in a RAID0. Which is it? I know benchmarking is hard, and it's difficult to test certain aspects of Amazon Timestream due to it being a managed service, but I really think these details need to be firmed up to make sure you are comparing apples to apples. TimescaleDB seems like a cool product. Another suggestion I'd make is to run the Amazon Timestream clients across multiple AZs if you aren't already. The blog post doesn't mention whether all the t3 instances are in the same AZ or not.
    - ryanbooz 6y ago
      > Another suggestion I'd make is to run the Amazon Timestream clients across multiple AZs if you aren't already. The blog post doesn't mention whether all the t3 instances are in the same AZ or not. They were all run in the same AZ. This brings up a good point, however, that we discussed internally when the first results came back. If there are tricks like this that might improve performance, it's not (currently) spelled out in the documentation so there's no way to know that. And we reached out for help in various forums with no response. It's worth noting that since we performed this analysis, Amazon did release their own tooling for a similar benchmark and created a post[0]. While neither it, nor the tooling documentation[1] specifically spell out how many threads or instances they ran to achieve their results, it's hard to draw an apples-to-apples comparison. It does reveal that they used (had to use??) an m5.24xlarge instance (96 vCPU,384GB) to run their tests. As discussed in the article, one much smaller t3 instance was able to add >1 million metrics/second into TimescaleDB running in Timescale Forge. [0] https://aws.amazon.com/blogs/database/deriving-real-time-insights-over-petabytes-of-time-series-data-with-amazon-timestream/ https://aws.amazon.com/blogs/database/deriving-real-time-ins... [1] https://github.com/awslabs/amazon-timestream-tools/tree/master/tools/perf-scale-workload https://github.com/awslabs/amazon-timestream-tools/tree/mast...
    - mfreed 6y ago
      It's raided network-attached block storage. We'll update the blog post to make that clear.

overcast 6y ago

Would that really explain the difference between 5 minutes on Timescale and 2 weeks on Timestream? Timeseries databases need the ability to ingest data rapidly.

ryanbooz 6y ago
Please see below for correction... but at some point today a poster said "~2 weeks" for Timestream... but that's incorrect. We ran the data load/ingest for "~2 days" (40 hours). Just want to make sure the right numbers are being used! Thanks! (Post author @Timescale)
- overcast 6y ago
  STILL! 5 minutes vs 2 DAYS can't be attributed to just being replicated.
  - rubiquity 6y ago
    Large numbers are unintuitive. If we’re conservative and add only 1 millisecond of latency for the 3 AZ replication, on 410 million records that is about 4 and a half days of time if the client is not doing any batching.
    - ryanbooz 6y ago
      TSBS did batch 100 metrics at a time, the max per request that Timestream allows. As we (and others) have suggested, this could easily be one of the reasons it's harder to ingest more quickly without significantly more threads. TimescaleDB (and other TSDB's) allow much higher batching values.