7 ms·

InfluxDB is now InfluxData, a platform for time series data

124 points by pauldix 11y ago

shubhra51 11y ago
Wohoo! Looks like the full stack.
Mahn 11y ago
I kind of prefered the old InfluxDB website but it may be a question of getting used to it. Either way I'm a fan, keep up the good work!
crudbug 11y ago
All the best to Influx team. I see a good platform strategy here.
raziel2p 11y ago
I'm a fan. Chronograf looks like an interesting alternative to Grafana - but it's not open source :( Kapacitor looks neat as well, though it's going up against stiff competition from Bosun.
- pauldix 11y ago
  Thanks! We'll definitely be open sourcing some things from Chronograf and it's free to use. We removed the registration part of it with the release we cut today. We're fans of Bosun and actually contributed code to the project to get it integrated with InfluxDB. However, we wanted something that was purpose built with the InfluxDB schema in mind, which doesn't map to any existing time series solution.
- fdelapena 11y ago
  Unfortunately, Kapacitor is script based. I'm looking forward to see the alert system Grafana is designing: https://github.com/grafana/grafana/issues/2209 https://github.com/grafana/grafana/issues/2209
plainOldText 11y ago
It's interesting to see some competition in this space. Basho, the maker of Riak has also moved from being a db company to being a data platform company. And they've also launched a time series version of Riak.
- pauldix 11y ago
  I did see the time series announcement from Basho and will follow it with interest :) How are they moving to be a data platform company?
  - plainOldText 11y ago
    http://basho.com/basho-data-platform/ http://basho.com/basho-data-platform/
jhgg 11y ago
Any thoughts on the viability of using this platform in place of say data dog or librato?
netinstructions 11y ago
I have been using InfluxDB (for throwing data into) and Grafana (for displaying pretty charts of that data). It's pretty great. I had no idea InfluxData forked Grafana and developed their own (currently closed-source) visualization tool named Chronograf. It looks like a InfluxData branded version of Grafana. Also interesting is that it's the only closed-source element of their TICK stack. Seems like there could easily be a conflict of interest here, such as new features of InfluxData only working with Chronograf and not Grafana. grafana - http://grafana.org/ http://grafana.org/ chronograf - https://influxdata.com/get-started/#visualizing-data-with-chronograf https://influxdata.com/get-started/#visualizing-data-with-ch...
- pauldix 11y ago
  We didn't fork Grafana. Chronograf is a completely new product built from the ground up. They don't use the same JavaScript technologies. Chronograf uses React and Dygraphs. I believe Grafana uses Angular and Flot. We're fans of Grafana and want InfluxDB to work well with it. However, we also want the ability to change things over time and possibly take a different focus. Over time we'll be pushing Chronograf to be usable for people that fall less and less on the development side of things. It's still early stage with Chronograf so we're still iterating and testing different ideas.
  - anonfunction 11y ago
    I recently switched from Grafana to Kibana due to the lack of different visualizations and sometimes data would not show up in the graphs, leaving blank spaces. The data was in influxdb. Nice to hear you're building your own project that gives you more freedom to build the best product you can.
    - pauldix 11y ago
      thanks!
  - netinstructions 11y ago
    If Chronograf is a completely new product kudos to your team. I assumed it was a fork of Grafana because it looked and acted very, very similar to Grafana based on the 'Getting Started' docs of Chronograf. Just curious, what are some reasons why one would choose Chronograf over Grafana right now?
lux 11y ago
First, I really like the rebrand and wider focus. I think it will help people understand how your suite of tools work together (I just started playing with InfluxDB + Go yesterday and am loving it, and Telegraf seems a natural fit for another upcoming project too). There's a lot of mention about the ephemeral nature of time series data, but how is InfluxDB for storing permanent time series data? Are there any performance limitations I should be thinking about (aside from the limits of the hardware itself)?
- cschneid 11y ago
  They've gone through a few engines. They have a post about the newest one which isn't quite out yet: https://influxdb.com/blog/2015/10/07/the_new_influxdb_storage_engine_a_time_structured_merge_tree.html https://influxdb.com/blog/2015/10/07/the_new_influxdb_storag... We use them for storing "permanent" data. They slice up the data into weekly files, so if you don't access it often, it's no biggie to store a lot of data. If you constantly access data across time, you may run into performance degradation from a caching point of view (having to pull in the files from past, blowing caches of new stuff constantly).
  - lux 11y ago
    Thanks, that's good to know!
- pauldix 11y ago
  InfluxDB is designed to either store data for some given period of time (a retention policy) or forever, whichever you need. The ephemeral nature of time series is more about the fact that metadata and the raw data are constantly changing. For example if you use a round robin database (like Graphite or RRD) you have to pre-allocate the file based on how much of the series data you're going to have (e.g. 10s samples for 3 months). Having series that are ephemeral and may exist for an hour or a day make this much harder to deal with. Our storage engine, the Time Structured Merge Tree, is designed with this ephemeral nature in mind. So it's able to store this data efficiently. The 0.9.6 release that came out today has that storage engine in it for testing. We'll release it for production use at the end of January after more testing.
lobster_johnson 11y ago
Their stack is starting to look good, at least on paper. I'm particularly interested in Telegraf. We're using Prometheus for data collection and monitoring right now, and collection system is definitely the weakest part.
- bbrazil 11y ago
  Prometheus developer here, what do you feel it's missing? Telegraf can both read and produce Prometheus metrics, which is something I've worked on as I believe metrics shouldn't be locked into any one ecosystem.
  - lobster_johnson 11y ago
    There's a bunch of things. First, Prometheus prefers that you create an HTTP API for metrics. This means having to manage running daemons, set aside ports, and so on. The fact that it's not pluggable leads to an ecosystem where every "exporter" is HTTP-based and needs to be run and maintained this way. It's not lightweight or ops-friendly. So for our own stuff, we built a single pluggable collector that simply emits text files (scheduled via cron), and uses prometheus-node-exporter's "textfile directory" support to export the data. It's not ideal. And since it's a kind of mini-framework itself, that doesn't follow any community-defined plugin system, we can't easily open-source the bits we think would be useful to others (we have some nice metrics for RabbitMQ, ElasticSearch, PostgreSQL, etc.) because people would have to buy into our little plugin system. I'd much rather that the exporter was pluggable, and was capable of spawning subprocesses. Prometheus also works very poorly/not at all with existing web frameworks such as Ruby's Unicorn and Node.js's cluster module, where there isn't a single process that can track metrics. So far, the attempts to accomplish this with Unicorn, for example, seem unsatisfactory.
lfittl 11y ago
The Influx team is doing great things - however from the perspective of someone who's tried using InfluxDB in the last few months for a production app: I really, really wish they'd hire 2-3 more database engineers to focus on the actual timeseries database, not nice looking tooling around it. Simple things like incremental backups are still not working as expected in recent releases. Useful features from the 0.8 series still haven't been ported. And tsm1 is a neverending effort it seems. We've went back to Postgres for storing timeseries data since we didn't feel confident to use InfluxDB in production.
- officialchicken 11y ago
  I haven't given up on Influx, yet, but I'm close. A simple "DELETE * FROM series" without a where clause the system to freeze up while the CPU spins out of control, usually requiring a reboot to clear up after the KILL'd proc.(0.9.4 + Ubuntu, FreeBSD, and Go 1.4/1.5 combos all exhibit the behavior for me). I have also not been able to get incremental backups working, have not been able to restore any snapshots, but I have had data survive reboots. Unless I can get fairly predictable behavior out of Influx the proceess on my systems, it's going to remain in the "unpredictable-so-undependable" category, and far away from "I should pay monthly for this service".
  - pauldix 11y ago
    DELTE FROM hasn't been wired up yet: https://github.com/influxdb/influxdb/issues/1647 https://github.com/influxdb/influxdb/issues/1647 Unfortunately, we were passing that through to the DB which was causing problems. The current release 0.9.6 will return an error. You can drop an entire series, but you can't yet drop a specific time range. We'll hopefully get that in soon, but we need to get TSM out in production before we start looking at those additional features.
- pauldix 11y ago
  We're actively hiring for database engineers right now. Backups are coming in the next release. Which features from 0.8 aren't there? TSM is an effort, but not never ending. We just refuse to put it out there until we've done enough testing on it to be certain about its robustness. We think the community will be happier if we wait to release it rather than rushing something out that may or may not be ready. That being said, please test out TSM on the 0.9.6 release. I think you'll find that its performance is better than anything we've released previously (including 0.8) and so far we've been able to write far more data into it than in previous 0.9 storage engines. I'm sorry to hear that you had to go back to Postgres for your time series storage. Hopefully our upcoming releases will give you a reason to take another look.
infinotize 11y ago
Good to see the forward progress. I needed to set up a metrics platform to serve tenants on a Mesos cluster, and picked InfluxDB over some alternatives for 3 main reasons: fairly reliable working state at the time (still using 0.8.8) with no dependencies, upward momentum, and stated priority of horizonal scaling. Looking back, it's held up reasonably well. I'm admittedly disappointed that stable, horizontally-scalable clustering keeps getting pushed to the next release. To be fair, it's hard, and the redisigns with the 0.9+ API and TSM tree are important - if you're going to sell people on a dedicated time series DB it has to be really really good at it. In the long run, the full ecosystem development here will pay off.

124 points by pauldix 11y ago

Wohoo! Looks like the full stack.

Mahn 11y ago

I kind of prefered the old InfluxDB website but it may be a question of getting used to it. Either way I'm a fan, keep up the good work!

crudbug 11y ago

All the best to Influx team. I see a good platform strategy here.

raziel2p 11y ago

I'm a fan. Chronograf looks like an interesting alternative to Grafana - but it's not open source :( Kapacitor looks neat as well, though it's going up against stiff competition from Bosun.

pauldix 11y ago
Thanks! We'll definitely be open sourcing some things from Chronograf and it's free to use. We removed the registration part of it with the release we cut today. We're fans of Bosun and actually contributed code to the project to get it integrated with InfluxDB. However, we wanted something that was purpose built with the InfluxDB schema in mind, which doesn't map to any existing time series solution.
fdelapena 11y ago
Unfortunately, Kapacitor is script based. I'm looking forward to see the alert system Grafana is designing: https://github.com/grafana/grafana/issues/2209 https://github.com/grafana/grafana/issues/2209

plainOldText 11y ago

It's interesting to see some competition in this space. Basho, the maker of Riak has also moved from being a db company to being a data platform company. And they've also launched a time series version of Riak.

pauldix 11y ago
I did see the time series announcement from Basho and will follow it with interest :) How are they moving to be a data platform company?
- plainOldText 11y ago
  http://basho.com/basho-data-platform/ http://basho.com/basho-data-platform/

jhgg 11y ago

Any thoughts on the viability of using this platform in place of say data dog or librato?

netinstructions 11y ago

I have been using InfluxDB (for throwing data into) and Grafana (for displaying pretty charts of that data). It's pretty great. I had no idea InfluxData forked Grafana and developed their own (currently closed-source) visualization tool named Chronograf. It looks like a InfluxData branded version of Grafana. Also interesting is that it's the only closed-source element of their TICK stack. Seems like there could easily be a conflict of interest here, such as new features of InfluxData only working with Chronograf and not Grafana. grafana - http://grafana.org/ http://grafana.org/ chronograf - https://influxdata.com/get-started/#visualizing-data-with-chronograf https://influxdata.com/get-started/#visualizing-data-with-ch...

pauldix 11y ago
We didn't fork Grafana. Chronograf is a completely new product built from the ground up. They don't use the same JavaScript technologies. Chronograf uses React and Dygraphs. I believe Grafana uses Angular and Flot. We're fans of Grafana and want InfluxDB to work well with it. However, we also want the ability to change things over time and possibly take a different focus. Over time we'll be pushing Chronograf to be usable for people that fall less and less on the development side of things. It's still early stage with Chronograf so we're still iterating and testing different ideas.
- anonfunction 11y ago
  I recently switched from Grafana to Kibana due to the lack of different visualizations and sometimes data would not show up in the graphs, leaving blank spaces. The data was in influxdb. Nice to hear you're building your own project that gives you more freedom to build the best product you can.
  - pauldix 11y ago
    thanks!
- netinstructions 11y ago
  If Chronograf is a completely new product kudos to your team. I assumed it was a fork of Grafana because it looked and acted very, very similar to Grafana based on the 'Getting Started' docs of Chronograf. Just curious, what are some reasons why one would choose Chronograf over Grafana right now?

lux 11y ago

First, I really like the rebrand and wider focus. I think it will help people understand how your suite of tools work together (I just started playing with InfluxDB + Go yesterday and am loving it, and Telegraf seems a natural fit for another upcoming project too). There's a lot of mention about the ephemeral nature of time series data, but how is InfluxDB for storing permanent time series data? Are there any performance limitations I should be thinking about (aside from the limits of the hardware itself)?

cschneid 11y ago
They've gone through a few engines. They have a post about the newest one which isn't quite out yet: https://influxdb.com/blog/2015/10/07/the_new_influxdb_storage_engine_a_time_structured_merge_tree.html https://influxdb.com/blog/2015/10/07/the_new_influxdb_storag... We use them for storing "permanent" data. They slice up the data into weekly files, so if you don't access it often, it's no biggie to store a lot of data. If you constantly access data across time, you may run into performance degradation from a caching point of view (having to pull in the files from past, blowing caches of new stuff constantly).
- lux 11y ago
  Thanks, that's good to know!
pauldix 11y ago
InfluxDB is designed to either store data for some given period of time (a retention policy) or forever, whichever you need. The ephemeral nature of time series is more about the fact that metadata and the raw data are constantly changing. For example if you use a round robin database (like Graphite or RRD) you have to pre-allocate the file based on how much of the series data you're going to have (e.g. 10s samples for 3 months). Having series that are ephemeral and may exist for an hour or a day make this much harder to deal with. Our storage engine, the Time Structured Merge Tree, is designed with this ephemeral nature in mind. So it's able to store this data efficiently. The 0.9.6 release that came out today has that storage engine in it for testing. We'll release it for production use at the end of January after more testing.

lobster_johnson 11y ago

Their stack is starting to look good, at least on paper. I'm particularly interested in Telegraf. We're using Prometheus for data collection and monitoring right now, and collection system is definitely the weakest part.

bbrazil 11y ago
Prometheus developer here, what do you feel it's missing? Telegraf can both read and produce Prometheus metrics, which is something I've worked on as I believe metrics shouldn't be locked into any one ecosystem.
- lobster_johnson 11y ago
  There's a bunch of things. First, Prometheus prefers that you create an HTTP API for metrics. This means having to manage running daemons, set aside ports, and so on. The fact that it's not pluggable leads to an ecosystem where every "exporter" is HTTP-based and needs to be run and maintained this way. It's not lightweight or ops-friendly. So for our own stuff, we built a single pluggable collector that simply emits text files (scheduled via cron), and uses prometheus-node-exporter's "textfile directory" support to export the data. It's not ideal. And since it's a kind of mini-framework itself, that doesn't follow any community-defined plugin system, we can't easily open-source the bits we think would be useful to others (we have some nice metrics for RabbitMQ, ElasticSearch, PostgreSQL, etc.) because people would have to buy into our little plugin system. I'd much rather that the exporter was pluggable, and was capable of spawning subprocesses. Prometheus also works very poorly/not at all with existing web frameworks such as Ruby's Unicorn and Node.js's cluster module, where there isn't a single process that can track metrics. So far, the attempts to accomplish this with Unicorn, for example, seem unsatisfactory.

lfittl 11y ago

The Influx team is doing great things - however from the perspective of someone who's tried using InfluxDB in the last few months for a production app: I really, really wish they'd hire 2-3 more database engineers to focus on the actual timeseries database, not nice looking tooling around it. Simple things like incremental backups are still not working as expected in recent releases. Useful features from the 0.8 series still haven't been ported. And tsm1 is a neverending effort it seems. We've went back to Postgres for storing timeseries data since we didn't feel confident to use InfluxDB in production.

officialchicken 11y ago
I haven't given up on Influx, yet, but I'm close. A simple "DELETE * FROM series" without a where clause the system to freeze up while the CPU spins out of control, usually requiring a reboot to clear up after the KILL'd proc.(0.9.4 + Ubuntu, FreeBSD, and Go 1.4/1.5 combos all exhibit the behavior for me). I have also not been able to get incremental backups working, have not been able to restore any snapshots, but I have had data survive reboots. Unless I can get fairly predictable behavior out of Influx the proceess on my systems, it's going to remain in the "unpredictable-so-undependable" category, and far away from "I should pay monthly for this service".
- pauldix 11y ago
  DELTE FROM hasn't been wired up yet: https://github.com/influxdb/influxdb/issues/1647 https://github.com/influxdb/influxdb/issues/1647 Unfortunately, we were passing that through to the DB which was causing problems. The current release 0.9.6 will return an error. You can drop an entire series, but you can't yet drop a specific time range. We'll hopefully get that in soon, but we need to get TSM out in production before we start looking at those additional features.
pauldix 11y ago
We're actively hiring for database engineers right now. Backups are coming in the next release. Which features from 0.8 aren't there? TSM is an effort, but not never ending. We just refuse to put it out there until we've done enough testing on it to be certain about its robustness. We think the community will be happier if we wait to release it rather than rushing something out that may or may not be ready. That being said, please test out TSM on the 0.9.6 release. I think you'll find that its performance is better than anything we've released previously (including 0.8) and so far we've been able to write far more data into it than in previous 0.9 storage engines. I'm sorry to hear that you had to go back to Postgres for your time series storage. Hopefully our upcoming releases will give you a reason to take another look.

infinotize 11y ago

Good to see the forward progress. I needed to set up a metrics platform to serve tenants on a Mesos cluster, and picked InfluxDB over some alternatives for 3 main reasons: fairly reliable working state at the time (still using 0.8.8) with no dependencies, upward momentum, and stated priority of horizonal scaling. Looking back, it's held up reasonably well. I'm admittedly disappointed that stable, horizontally-scalable clustering keeps getting pushed to the next release. To be fair, it's hard, and the redisigns with the 0.9+ API and TSM tree are important - if you're going to sell people on a dedicated time series DB it has to be really really good at it. In the long run, the full ecosystem development here will pay off.