3 ms·

First, I really like the rebrand and wider focus. I think it will help people understand how your suite of tools work together (I just started playing with Infl

by lux 11y ago

First, I really like the rebrand and wider focus. I think it will help people understand how your suite of tools work together (I just started playing with InfluxDB + Go yesterday and am loving it, and Telegraf seems a natural fit for another upcoming project too). There's a lot of mention about the ephemeral nature of time series data, but how is InfluxDB for storing permanent time series data? Are there any performance limitations I should be thinking about (aside from the limits of the hardware itself)?

cschneid 11y ago
They've gone through a few engines. They have a post about the newest one which isn't quite out yet: https://influxdb.com/blog/2015/10/07/the_new_influxdb_storage_engine_a_time_structured_merge_tree.html https://influxdb.com/blog/2015/10/07/the_new_influxdb_storag... We use them for storing "permanent" data. They slice up the data into weekly files, so if you don't access it often, it's no biggie to store a lot of data. If you constantly access data across time, you may run into performance degradation from a caching point of view (having to pull in the files from past, blowing caches of new stuff constantly).
- lux 11y ago
  Thanks, that's good to know!
pauldix 11y ago
InfluxDB is designed to either store data for some given period of time (a retention policy) or forever, whichever you need. The ephemeral nature of time series is more about the fact that metadata and the raw data are constantly changing. For example if you use a round robin database (like Graphite or RRD) you have to pre-allocate the file based on how much of the series data you're going to have (e.g. 10s samples for 3 months). Having series that are ephemeral and may exist for an hour or a day make this much harder to deal with. Our storage engine, the Time Structured Merge Tree, is designed with this ephemeral nature in mind. So it's able to store this data efficiently. The 0.9.6 release that came out today has that storage engine in it for testing. We'll release it for production use at the end of January after more testing.
- lux 11y ago
  Thanks for the explanation. InfluxDB has been such a pleasure to work with so far that I wanted to make sure I wasn't missing something before getting too invested :)

by lux 11y ago

cschneid 11y ago

They've gone through a few engines. They have a post about the newest one which isn't quite out yet: https://influxdb.com/blog/2015/10/07/the_new_influxdb_storage_engine_a_time_structured_merge_tree.html https://influxdb.com/blog/2015/10/07/the_new_influxdb_storag... We use them for storing "permanent" data. They slice up the data into weekly files, so if you don't access it often, it's no biggie to store a lot of data. If you constantly access data across time, you may run into performance degradation from a caching point of view (having to pull in the files from past, blowing caches of new stuff constantly).

lux 11y ago
Thanks, that's good to know!

pauldix 11y ago

InfluxDB is designed to either store data for some given period of time (a retention policy) or forever, whichever you need. The ephemeral nature of time series is more about the fact that metadata and the raw data are constantly changing. For example if you use a round robin database (like Graphite or RRD) you have to pre-allocate the file based on how much of the series data you're going to have (e.g. 10s samples for 3 months). Having series that are ephemeral and may exist for an hour or a day make this much harder to deal with. Our storage engine, the Time Structured Merge Tree, is designed with this ephemeral nature in mind. So it's able to store this data efficiently. The 0.9.6 release that came out today has that storage engine in it for testing. We'll release it for production use at the end of January after more testing.

lux 11y ago
Thanks for the explanation. InfluxDB has been such a pleasure to work with so far that I wanted to make sure I wasn't missing something before getting too invested :)