7 ms·

Amazon Timestream is generally available

91 points by dsflora 6y ago

kondro 6y ago
Only 2 years since the original announcement. :D A bit strange that only memory & magnetic stores are currently available. I'm curious as to why the SSD option isn't available yet.
- hbcondo714 6y ago
  Yeah we signed up for the beta shortly after that announcement but never heard back from AWS except an automated reply. We even asked our account rep this summer as to the status of timestream and he had no idea. Now it's GA!
  - kondro 6y ago
    We didn't hear back until 4 months ago… and only after continually pestering our account manager.
- chrisjc 6y ago
  Doesn't magnetic make more sense since I assume the data is laid down (timeseries) in a serial fashion?
  - valyala 6y ago
    SSD may help in the case when a query needs to process data from many time series over big time range. Data for a single time series can be laid down in a serial fashion. When reading data from many time series, then many disk seeks are usually needed for jumping between time series. This may be optimized somehow for common case when the query touches related time series. Such optimizations are implemented in VictoriaMetrics [1], so it usually outperforms other time series databases on high-latency magnetic disks with low IOPS. [1] https://victoriametrics.github.io/ https://victoriametrics.github.io/
paragraft 6y ago
The pricing page seems off, $0.03 per GB-hour for the magnetic store, while the unreleased SSD store is $0.0004167 per GB-hour? Comparing with S3 prices surely the magnetic store is closer to $0.03 per GB-month?
- deleted 6y ago
  [deleted]
- jeremycarter 6y ago
  Yeah I would expect the magnetic to be $0.00035
- kondro 6y ago
  I'm pretty sure that is $0.03/GB/month (rated hourly) for magnetic storage as per the example. It's just written in a confusing way.
  - paragraft 6y ago
    I saw the examples, but that was even more confusing because I think the maths is wrong on almost all the calcs: From the example: "Writes cost: $ 8.64 per month. This is computed as: ( 2 writes * 100 EC2 instances * 60 minutes * 24 hours * 30 days) * $0.50/ 1 MM writes" I get $4.32 from ((2 * 100 * 60 * 24 * 30) * 0.5) / 1M From the example: "Memory store cost: $ 3.55 per month. This is computed as: ( 2 KB per minute per instance * 100 instances * 60 minutes * 6 hours * 30 days) * $0.036 / GB-hour." For starters I think there needs to be another "* 24 hours" in there - the 6 hours is to account for the retention period, but they're storing that 24 hours a day... But even with that I get $1.87 for the memory store from ((2 * 100 * 60 * 6 * 24 * 30) / 1000000) * 0.036 (the div by 1M is to convert KB to GB) From the example: "Magnetic store cost: $ 5.93 per month. This is computed as: (2 KB per minute per instance * 100 instances * 60 minutes * 24 hours * 30 days * 12 months) * $0.03 /GB-month." But I get $3.11: ((2 * 100 * 60 * 24 * 30 * 12) / 1000000) * 0.03 From the example: "Query cost (alerting queries): $ 7.03 per month. This is computed as (10 MB per alerting query * 100 queries per hour * 24 hours * 30 days * $0.01 / GB). Alerting queries process 5 minutes of data. 5 minutes of data is 1 MB (2 KB per minute per instance * 100 instances * 5), which gets rounded up to 10 MB minimum charge for queries." But I get $7.20 from (10 * 100 * 24 * 30 * 0.01) / 1000 (MB -> GB) It's only on the "Query cost (ad-hoc queries)" that I can make the answer agree. (They have now updated the magnetic price, as noted by others)
comboy 6y ago
From my limited experience with time series, not being able to insert data older than X seems like a big disadvantage that may bite later down the road. Bugs happen.
- chrisdalke 6y ago
  Was just going to post the same, this seems great for rolling analytics on software or VMs, as a backing for monitoring infrastructure and pairing with Grafana. However for IoT or hardware time series I’m left with a few questions... A few things it seems are not possible: - An IoT device with limited network connection that batches data points while offline - manually loading time series data from disk at a later date on an offline device such as a weather station - loading data “in the future” (data points with bad/erratic timestamps, memoized predictions) I guess the hardware / IoT use case is pretty small compared to infrastructure monitoring, where Timestream’s design choices make a lot of sense.
  - foxrob92 6y ago
    >I guess the hardware / IoT use case is pretty small OSIsoft was bought earlier this year by AVEVA for $5 billion[1]. OSIsoft's offering is centered around storing industrial control system data (the buzzword-version is IIoT, though that is a bit more broad). They're also not the only player in this space. Many of the industrial automation vendors also have their own offerings. [1] https://www.bloomberg.com/news/articles/2020-08-25/aveva-to-buy-industrial-software-firm-osisoft-in-5-billion-deal https://www.bloomberg.com/news/articles/2020-08-25/aveva-to-...
  - valyala 6y ago
    Take a look at vmagent [1]. It has been designed to run and collect monitoring data on IoT and edge devices and then upload it to remote storage in short time frames when connection to the storage can be established. There are remote storage systems such as VictoriaMetrics, which support loading historical data out of the box [2]. [1] https://victoriametrics.github.io/vmagent.html https://victoriametrics.github.io/vmagent.html [2] https://victoriametrics.github.io/#backfilling https://victoriametrics.github.io/#backfilling
sdan 6y ago
AWS looked at Timescale's new license and named their db Timestream. Haha, classic.
- akulkarni 6y ago
  Yes, and we (Timescale) are flattered ;-) Unfortunately for Timestream, Timescale Cloud is 10x-70x cheaper to use than Timestream. [0] Timescale Cloud is also in 76 regions across AWS, Azure, and Google. Timestream only in 4 regions, 1 cloud. [1] [0] https://docs.google.com/spreadsheets/d/1Nb9wTLqlWB_uch_VKuImSgSqmsccQMBcHPWlHFVmh28/edit?ts=5ef3b400#gid=0 https://docs.google.com/spreadsheets/d/1Nb9wTLqlWB_uch_VKuIm... [1] https://blog.timescale.com/blog/fully-managed-time-series-data-service-now-available-in-aws-azure-gcp-75-regions-compare-vs-influxdb-timestream/ https://blog.timescale.com/blog/fully-managed-time-series-da...
  - deleted 6y ago
    [deleted]
- kondro 6y ago
  Well, in AWS' defence, Timestream was announced at re:invent in 2018.
  - akulkarni 6y ago
    Timescale was announced in April 2017. The name isn't a coincidence. :-) https://blog.timescale.com/blog/when-boring-is-awesome-building-a-scalable-time-series-database-on-postgresql-2900ea453ee2/ https://blog.timescale.com/blog/when-boring-is-awesome-build...
aratno 6y ago
I see some references to Presto, but does anyone know what this engine is based on? Seems vaguely similar to Druid but it's hard to tell.
jeremycarter 6y ago
We use Timescale for IoT telemetry but we also are fully in GCP. I wonder if GCP are also planning a fully fledged timeseries database?
- akulkarni 6y ago
  (Timescale person) Thanks for using the service. Feel free to send us any product feedback, we are all ears :-)
  - jeremycarter 6y ago
    Keep up the great work :)
    - akulkarni 6y ago
      Thank you :-) Will share with the rest of the team!
nnx 6y ago
Despite 2 years of preview, this GA launch looks like the most half-baked I've seen from AWS. No SSD Store? Only available in US regions? Pricing page does not have the usual per-region dropdown and instead says "he pricing in Europe (Ireland) Region will be 13.1% more than the price below." edit: also just noticed the Go SDK is uniquely split into 2 packages for some reason "service/timestream-query" and "service/timestream-write".
foreigner 6y ago
200 years LOL. That's some impressive hubris.

91 points by dsflora 6y ago

Only 2 years since the original announcement. :D A bit strange that only memory & magnetic stores are currently available. I'm curious as to why the SSD option isn't available yet.

hbcondo714 6y ago
Yeah we signed up for the beta shortly after that announcement but never heard back from AWS except an automated reply. We even asked our account rep this summer as to the status of timestream and he had no idea. Now it's GA!
- kondro 6y ago
  We didn't hear back until 4 months ago… and only after continually pestering our account manager.
chrisjc 6y ago
Doesn't magnetic make more sense since I assume the data is laid down (timeseries) in a serial fashion?
- valyala 6y ago
  SSD may help in the case when a query needs to process data from many time series over big time range. Data for a single time series can be laid down in a serial fashion. When reading data from many time series, then many disk seeks are usually needed for jumping between time series. This may be optimized somehow for common case when the query touches related time series. Such optimizations are implemented in VictoriaMetrics [1], so it usually outperforms other time series databases on high-latency magnetic disks with low IOPS. [1] https://victoriametrics.github.io/ https://victoriametrics.github.io/

paragraft 6y ago

The pricing page seems off, $0.03 per GB-hour for the magnetic store, while the unreleased SSD store is $0.0004167 per GB-hour? Comparing with S3 prices surely the magnetic store is closer to $0.03 per GB-month?

deleted 6y ago
[deleted]
jeremycarter 6y ago
Yeah I would expect the magnetic to be $0.00035
kondro 6y ago
I'm pretty sure that is $0.03/GB/month (rated hourly) for magnetic storage as per the example. It's just written in a confusing way.
- paragraft 6y ago
  I saw the examples, but that was even more confusing because I think the maths is wrong on almost all the calcs: From the example: "Writes cost: $ 8.64 per month. This is computed as: ( 2 writes * 100 EC2 instances * 60 minutes * 24 hours * 30 days) * $0.50/ 1 MM writes" I get $4.32 from ((2 * 100 * 60 * 24 * 30) * 0.5) / 1M From the example: "Memory store cost: $ 3.55 per month. This is computed as: ( 2 KB per minute per instance * 100 instances * 60 minutes * 6 hours * 30 days) * $0.036 / GB-hour." For starters I think there needs to be another "* 24 hours" in there - the 6 hours is to account for the retention period, but they're storing that 24 hours a day... But even with that I get $1.87 for the memory store from ((2 * 100 * 60 * 6 * 24 * 30) / 1000000) * 0.036 (the div by 1M is to convert KB to GB) From the example: "Magnetic store cost: $ 5.93 per month. This is computed as: (2 KB per minute per instance * 100 instances * 60 minutes * 24 hours * 30 days * 12 months) * $0.03 /GB-month." But I get $3.11: ((2 * 100 * 60 * 24 * 30 * 12) / 1000000) * 0.03 From the example: "Query cost (alerting queries): $ 7.03 per month. This is computed as (10 MB per alerting query * 100 queries per hour * 24 hours * 30 days * $0.01 / GB). Alerting queries process 5 minutes of data. 5 minutes of data is 1 MB (2 KB per minute per instance * 100 instances * 5), which gets rounded up to 10 MB minimum charge for queries." But I get $7.20 from (10 * 100 * 24 * 30 * 0.01) / 1000 (MB -> GB) It's only on the "Query cost (ad-hoc queries)" that I can make the answer agree. (They have now updated the magnetic price, as noted by others)

comboy 6y ago

From my limited experience with time series, not being able to insert data older than X seems like a big disadvantage that may bite later down the road. Bugs happen.

chrisdalke 6y ago
Was just going to post the same, this seems great for rolling analytics on software or VMs, as a backing for monitoring infrastructure and pairing with Grafana. However for IoT or hardware time series I’m left with a few questions... A few things it seems are not possible: - An IoT device with limited network connection that batches data points while offline - manually loading time series data from disk at a later date on an offline device such as a weather station - loading data “in the future” (data points with bad/erratic timestamps, memoized predictions) I guess the hardware / IoT use case is pretty small compared to infrastructure monitoring, where Timestream’s design choices make a lot of sense.
- foxrob92 6y ago
  >I guess the hardware / IoT use case is pretty small OSIsoft was bought earlier this year by AVEVA for $5 billion[1]. OSIsoft's offering is centered around storing industrial control system data (the buzzword-version is IIoT, though that is a bit more broad). They're also not the only player in this space. Many of the industrial automation vendors also have their own offerings. [1] https://www.bloomberg.com/news/articles/2020-08-25/aveva-to-buy-industrial-software-firm-osisoft-in-5-billion-deal https://www.bloomberg.com/news/articles/2020-08-25/aveva-to-...
- valyala 6y ago
  Take a look at vmagent [1]. It has been designed to run and collect monitoring data on IoT and edge devices and then upload it to remote storage in short time frames when connection to the storage can be established. There are remote storage systems such as VictoriaMetrics, which support loading historical data out of the box [2]. [1] https://victoriametrics.github.io/vmagent.html https://victoriametrics.github.io/vmagent.html [2] https://victoriametrics.github.io/#backfilling https://victoriametrics.github.io/#backfilling

sdan 6y ago

AWS looked at Timescale's new license and named their db Timestream. Haha, classic.

akulkarni 6y ago
Yes, and we (Timescale) are flattered ;-) Unfortunately for Timestream, Timescale Cloud is 10x-70x cheaper to use than Timestream. [0] Timescale Cloud is also in 76 regions across AWS, Azure, and Google. Timestream only in 4 regions, 1 cloud. [1] [0] https://docs.google.com/spreadsheets/d/1Nb9wTLqlWB_uch_VKuImSgSqmsccQMBcHPWlHFVmh28/edit?ts=5ef3b400#gid=0 https://docs.google.com/spreadsheets/d/1Nb9wTLqlWB_uch_VKuIm... [1] https://blog.timescale.com/blog/fully-managed-time-series-data-service-now-available-in-aws-azure-gcp-75-regions-compare-vs-influxdb-timestream/ https://blog.timescale.com/blog/fully-managed-time-series-da...
- deleted 6y ago
  [deleted]
kondro 6y ago
Well, in AWS' defence, Timestream was announced at re:invent in 2018.
- akulkarni 6y ago
  Timescale was announced in April 2017. The name isn't a coincidence. :-) https://blog.timescale.com/blog/when-boring-is-awesome-building-a-scalable-time-series-database-on-postgresql-2900ea453ee2/ https://blog.timescale.com/blog/when-boring-is-awesome-build...

aratno 6y ago

I see some references to Presto, but does anyone know what this engine is based on? Seems vaguely similar to Druid but it's hard to tell.

jeremycarter 6y ago

We use Timescale for IoT telemetry but we also are fully in GCP. I wonder if GCP are also planning a fully fledged timeseries database?

akulkarni 6y ago
(Timescale person) Thanks for using the service. Feel free to send us any product feedback, we are all ears :-)
- jeremycarter 6y ago
  Keep up the great work :)
  - akulkarni 6y ago
    Thank you :-) Will share with the rest of the team!

nnx 6y ago

Despite 2 years of preview, this GA launch looks like the most half-baked I've seen from AWS. No SSD Store? Only available in US regions? Pricing page does not have the usual per-region dropdown and instead says "he pricing in Europe (Ireland) Region will be 13.1% more than the price below." edit: also just noticed the Go SDK is uniquely split into 2 packages for some reason "service/timestream-query" and "service/timestream-write".

foreigner 6y ago

200 years LOL. That's some impressive hubris.