4 ms·

From my limited experience with time series, not being able to insert data older than X seems like a big disadvantage that may bite later down the road. Bugs ha

by comboy 6y ago

From my limited experience with time series, not being able to insert data older than X seems like a big disadvantage that may bite later down the road. Bugs happen.

chrisdalke 6y ago
Was just going to post the same, this seems great for rolling analytics on software or VMs, as a backing for monitoring infrastructure and pairing with Grafana. However for IoT or hardware time series I’m left with a few questions... A few things it seems are not possible: - An IoT device with limited network connection that batches data points while offline - manually loading time series data from disk at a later date on an offline device such as a weather station - loading data “in the future” (data points with bad/erratic timestamps, memoized predictions) I guess the hardware / IoT use case is pretty small compared to infrastructure monitoring, where Timestream’s design choices make a lot of sense.
- foxrob92 6y ago
  >I guess the hardware / IoT use case is pretty small OSIsoft was bought earlier this year by AVEVA for $5 billion[1]. OSIsoft's offering is centered around storing industrial control system data (the buzzword-version is IIoT, though that is a bit more broad). They're also not the only player in this space. Many of the industrial automation vendors also have their own offerings. [1] https://www.bloomberg.com/news/articles/2020-08-25/aveva-to-buy-industrial-software-firm-osisoft-in-5-billion-deal https://www.bloomberg.com/news/articles/2020-08-25/aveva-to-...
- valyala 6y ago
  Take a look at vmagent [1]. It has been designed to run and collect monitoring data on IoT and edge devices and then upload it to remote storage in short time frames when connection to the storage can be established. There are remote storage systems such as VictoriaMetrics, which support loading historical data out of the box [2]. [1] https://victoriametrics.github.io/vmagent.html https://victoriametrics.github.io/vmagent.html [2] https://victoriametrics.github.io/#backfilling https://victoriametrics.github.io/#backfilling
foxrob92 6y ago
Yeah, I currently work for a company that does a time series DB for industrial applications (think power generation, mining, etc). Backfilling/inserting old data is common. Control network connectivity is not always great. Sometimes you need to do fill in old data. Forecasts are also a common use case (e.g. forecasting grid demand/capacity), so not being able to handle future data is another limitation.
- jeremycarter 6y ago
  Yeah we're not talking about buggy telemetry publishing code, we're talking about systems that could have poor connectivity that can only publish data once every 10 days (but include all samples). Seems like a massive oversight.
  - foxrob92 6y ago
    The other surprising factor is that a lot of heavy industry will have data going back decades. If they can't get all of that data in to timestream, that's a massive mark against it. I'd heard that it was initially developed for a major industrial operator (sounded like it was in the resources industry though I never found out who it was for), so the lack of support for late arriving data is a big surprise. That being said, there aren't any of those companies on the customers page - maybe it was just an incorrect rumour.
- valyala 6y ago
  BTW, which solution do you use for this case? Did you look at vmagent? It supports the case for collecting data from industrial appliances and uploading it to a central time series database when connection to it can be established [1]. [1] https://victoriametrics.github.io/vmagent.html#use-cases https://victoriametrics.github.io/vmagent.html#use-cases
kondro 6y ago
You can insert old data up to the in-memory retention period you choose (which can be 1 - 8766 hours). You can't insert future data more than 15 minutes into the future though.
- foxrob92 6y ago
  Sure, but if you configure your database for, lets say, 4 hours of in-memory, and you have a network outage of 12 hours (this is not uncommon on mine sites), you lose 8h of data and cannot put it in to your "master store".
  - kondro 6y ago
    If you expect a 12 hour outage, you should probably spec for 12 hours. Memory-based storage is expensive, but not ruinously so. The reason why this needs to be in-memory (I'm guessing) is a combination of dedupe checks and efficient ordered logs in the on-disk journals. Additionally, you can scale the memory retention policy up and down as you see fit. So if you notice a network outage, you can scale up the retention policy for the duration and then reduce it back down again after.
    - foxrob92 6y ago
      >If you expect a 12 hour outage, you should probably spec for 12 hours. Industrial facilities don't always expect their outages. > Additionally, you can scale the memory retention policy up and down as you see fit. So if you notice a network outage, you can scale up the retention policy for the duration and then reduce it back down again after. Better, though still a hassle to manage.
abraxas 6y ago
Do they have no historical backfill at all?
debug-desperado 6y ago
I believe Timescale has a similar issue if you’ve enabled compression (and it’s usually too expensive not to). Perhaps the main reason we’re sticking with Influx for the moment.
- akulkarni 6y ago
  (Timescale person) You can backfill or update data that lives in a compressed chunk by first decompressing that chunk. [0] But we are working on making it easier. :-) [0] https://docs.timescale.com/latest/using-timescaledb/compression#decompress-chunks https://docs.timescale.com/latest/using-timescaledb/compress...
  - jeremycarter 6y ago
    We have compression enabled for data older than 3 months. This gives us enough wiggle room to insert/modify anything recent.
- valyala 6y ago
  Take a look also at VictoriaMetrics - it supports loading historical data without any restrictions [1]. It supports Influx line protocol for data ingestion [2] and it usually requires lower amounts of RAM / CPU / disk space comparing to InfluxDB [3]. [1] https://victoriametrics.github.io/#backfilling https://victoriametrics.github.io/#backfilling [2] https://victoriametrics.github.io/#how-to-import-time-series-data https://victoriametrics.github.io/#how-to-import-time-series... [3] https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893 https://medium.com/@valyala/insert-benchmarks-with-inch-infl...

by comboy 6y ago

From my limited experience with time series, not being able to insert data older than X seems like a big disadvantage that may bite later down the road. Bugs happen.

chrisdalke 6y ago

Was just going to post the same, this seems great for rolling analytics on software or VMs, as a backing for monitoring infrastructure and pairing with Grafana. However for IoT or hardware time series I’m left with a few questions... A few things it seems are not possible: - An IoT device with limited network connection that batches data points while offline - manually loading time series data from disk at a later date on an offline device such as a weather station - loading data “in the future” (data points with bad/erratic timestamps, memoized predictions) I guess the hardware / IoT use case is pretty small compared to infrastructure monitoring, where Timestream’s design choices make a lot of sense.

foxrob92 6y ago
>I guess the hardware / IoT use case is pretty small OSIsoft was bought earlier this year by AVEVA for $5 billion[1]. OSIsoft's offering is centered around storing industrial control system data (the buzzword-version is IIoT, though that is a bit more broad). They're also not the only player in this space. Many of the industrial automation vendors also have their own offerings. [1] https://www.bloomberg.com/news/articles/2020-08-25/aveva-to-buy-industrial-software-firm-osisoft-in-5-billion-deal https://www.bloomberg.com/news/articles/2020-08-25/aveva-to-...
valyala 6y ago
Take a look at vmagent [1]. It has been designed to run and collect monitoring data on IoT and edge devices and then upload it to remote storage in short time frames when connection to the storage can be established. There are remote storage systems such as VictoriaMetrics, which support loading historical data out of the box [2]. [1] https://victoriametrics.github.io/vmagent.html https://victoriametrics.github.io/vmagent.html [2] https://victoriametrics.github.io/#backfilling https://victoriametrics.github.io/#backfilling

foxrob92 6y ago

Yeah, I currently work for a company that does a time series DB for industrial applications (think power generation, mining, etc). Backfilling/inserting old data is common. Control network connectivity is not always great. Sometimes you need to do fill in old data. Forecasts are also a common use case (e.g. forecasting grid demand/capacity), so not being able to handle future data is another limitation.

jeremycarter 6y ago
Yeah we're not talking about buggy telemetry publishing code, we're talking about systems that could have poor connectivity that can only publish data once every 10 days (but include all samples). Seems like a massive oversight.
- foxrob92 6y ago
  The other surprising factor is that a lot of heavy industry will have data going back decades. If they can't get all of that data in to timestream, that's a massive mark against it. I'd heard that it was initially developed for a major industrial operator (sounded like it was in the resources industry though I never found out who it was for), so the lack of support for late arriving data is a big surprise. That being said, there aren't any of those companies on the customers page - maybe it was just an incorrect rumour.
valyala 6y ago
BTW, which solution do you use for this case? Did you look at vmagent? It supports the case for collecting data from industrial appliances and uploading it to a central time series database when connection to it can be established [1]. [1] https://victoriametrics.github.io/vmagent.html#use-cases https://victoriametrics.github.io/vmagent.html#use-cases

kondro 6y ago

You can insert old data up to the in-memory retention period you choose (which can be 1 - 8766 hours). You can't insert future data more than 15 minutes into the future though.

foxrob92 6y ago
Sure, but if you configure your database for, lets say, 4 hours of in-memory, and you have a network outage of 12 hours (this is not uncommon on mine sites), you lose 8h of data and cannot put it in to your "master store".
- kondro 6y ago
  If you expect a 12 hour outage, you should probably spec for 12 hours. Memory-based storage is expensive, but not ruinously so. The reason why this needs to be in-memory (I'm guessing) is a combination of dedupe checks and efficient ordered logs in the on-disk journals. Additionally, you can scale the memory retention policy up and down as you see fit. So if you notice a network outage, you can scale up the retention policy for the duration and then reduce it back down again after.
  - foxrob92 6y ago
    >If you expect a 12 hour outage, you should probably spec for 12 hours. Industrial facilities don't always expect their outages. > Additionally, you can scale the memory retention policy up and down as you see fit. So if you notice a network outage, you can scale up the retention policy for the duration and then reduce it back down again after. Better, though still a hassle to manage.

abraxas 6y ago

Do they have no historical backfill at all?

debug-desperado 6y ago

I believe Timescale has a similar issue if you’ve enabled compression (and it’s usually too expensive not to). Perhaps the main reason we’re sticking with Influx for the moment.

akulkarni 6y ago
(Timescale person) You can backfill or update data that lives in a compressed chunk by first decompressing that chunk. [0] But we are working on making it easier. :-) [0] https://docs.timescale.com/latest/using-timescaledb/compression#decompress-chunks https://docs.timescale.com/latest/using-timescaledb/compress...
- jeremycarter 6y ago
  We have compression enabled for data older than 3 months. This gives us enough wiggle room to insert/modify anything recent.
valyala 6y ago
Take a look also at VictoriaMetrics - it supports loading historical data without any restrictions [1]. It supports Influx line protocol for data ingestion [2] and it usually requires lower amounts of RAM / CPU / disk space comparing to InfluxDB [3]. [1] https://victoriametrics.github.io/#backfilling https://victoriametrics.github.io/#backfilling [2] https://victoriametrics.github.io/#how-to-import-time-series-data https://victoriametrics.github.io/#how-to-import-time-series... [3] https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893 https://medium.com/@valyala/insert-benchmarks-with-inch-infl...