7 ms·

InfluxDB 1.1: 60% performance increase and new query functionality

121 points by otoolep 10y ago

betimd 10y ago
Very good to hear that. Started to try influxdb as my tsdb for iot and looks very promising.
ckdarby 10y ago
Currently using internally for production level work. They need to spend serious time making it easier to diagnose issues when they happen. The log is simply not verbose enough and when I hit issues with telegram, kapacitor and influxdb I have to open up a github issue having no idea what is causing the issue. Still highly recommended they just need to spend time making it easier to identify common problems. It'll help reduce the number of github issues they have opened too.
chrissnell 10y ago
I'm looking forward to trying it out. I used it at home to build a live weather site for my backyard weather station: https://mhkweather.com https://mhkweather.com The performance went way down when I moved my DB from a ext4 SSD volume to a four-disk ZFS magnetic volume. I expected some slowdown but not as much as I saw. Looking forward to trying 1.1 to see if it improves things.
otoolep 10y ago
Nice work by the InfluxDB team.
regecks 10y ago
Sweet. I had some 'getting oomkilled' problems with boundless queries back when I started using Influx for production early in the year, but it has been rock solid and performant since the later 0.1x releases. > Support for regular expressions on field keys in the SELECT clause. For example, SELECT /cpu_\d/ FROM cpu Woohoo! Very nice for making Grafana dashboards. > The admin UI (port 8083) is now officially deprecated and disabled in this release Ahh. I must be the only one using it, it gave kind of a good overview of all your collections. A bit of a pain to do from Grafana but oh well.
- CSDude 10y ago
  The admin UI was really good and they started to strip features away from it one by one and now it is disabled. It even could do simple graphs and was very useful for debugging.
  - zenlot 10y ago
    That seems to be their uncommunicated plan - to attract users to open source version and then remove features to offer just with enterprise product. They have done it with clustering support where after attracting people with promises they then removed it from open source version. Grafana as a replacement? It really does different things.
    - omn1 10y ago
      That sounds a bit unfair to me. In the end it's a company and they need to earn money somehow. Clustering is a very hard problem that takes a lot of manpower to get right.
      - imglorp 10y ago
        They do deserve to be paid for their product, yes. And it's worth paying for. They ill will was also deserved though, because when they changed business plans to exclude more features from the community version, they surprised and angered many users. Bait and switch is what got communicated, and trust violated, forcing projects to seek alternatives or modify their budgets. This can be avoided by clearly communicating the roadmap of community vs enterprise features and sticking to it. But don't keep danging new features in free and then remove them unexpectedly.
    - pauldix 10y ago
      The admin UI functionality is getting moved over to Chronograf, which is now open source. We do have commercial stuff, but trying to make money off our UI was never a strategy.
- omn1 10y ago
c2h5oh 10y ago
Tried using in production, failed really badly performance-wise (granted, it wasn't a good fit - cardinality was high). Replaced with PostgreSQL and I'm seeing 2+ orders of magnitude improvement on c4.large vs r4.8xlarge before..
- scrollaway 10y ago
  Sounds like you were using the wrong tool for the job. Influx is not for high cardinality data.
  - hbs 10y ago
    The problem is the definition of high cardinality, the Gorilla paper talks about 2B metrics at FB but the hardware sizing page for influx shows a scary graph which basically kills any series cardinality above a million https://docs.influxdata.com/influxdb/v1.1/guides/hardware_sizing/ https://docs.influxdata.com/influxdb/v1.1/guides/hardware_si... quite a difference!
    - user5994461 10y ago
      InfluxData is a single server app with zero failover and zero sharding. There is no "billion" metrics going in there. The architecture (or lack thereof) is not intended for that. You can try vertical scaling (if you're not in the cloud), you'll squeeze more metrics until you overflow your network cards or your hard drives or your CPU.
- agentgt 10y ago
  In the same boat. I tried Druid as well which was almost a better solution but lacked the flexibility and maturity. We are using Postgres + citus + some HLL. Our success is probably because we just have a lot more experience with Postgres. I still haven't tried HBase though.
daenney 10y ago
Can anyone verify the claims on their production setup? I'm always a bit weary of these announcements. It's easier to get big numbers like that on benchmarks than in real usage scenarios, has usually been my experience.
siscia 10y ago
I am wondering the choice of go as language to build a database. I am a big fan of golang for some workload but the GC will kill performance in a database, wouldn't? I am not being a brat I am seriously wondering how much GC affect the overall performance.
- rakoo 10y ago
  Then you should be happy that Go has been putting a lot of effort into making its GC as non-blocking as possible. There was an interesting blog post from Pusher yesterday (https://blog.pusher.com/golangs-real-time-gc-in-theory-and-practice/ https://blog.pusher.com/golangs-real-time-gc-in-theory-and-p...) about how they had to learn how the GC works and how to make the most of it; the links give more information into golang's internal evolution.
- maaaats 10y ago
  > but the GC will kill performance in a database, wouldn't? Why do you assume it would?
- daenney 10y ago
  > I am a big fan of golang for some workload but the GC will kill performance in a database, wouldn't? Cassandra and HSQLDB are written in Java for example which usually suffer from much larger GC collections/pauses (though you can read a book or two on the subject and tune it). So is Neo4j, a graph database. Or Datomic in Clojure (which runs on the JVM). Then there's Mnesia written in Erlang. A time series database, or a graph database, is quite different from a more general purpose database like Postgres/MySQL and means you need to consider different things. GC isn't necessarily an issue for high throughput (read or write). A lot of it depends on how much pressure you're putting on the GC (this is true for any type of application). Being aware of how your chosen language's GC behaves paired with the (typical) usage/behaviour of your code is probably the more important thing to consider. Just because something is written in a language without a GC doesn't automatically make it performant either.
  - daenney 10y ago
    Oh, and about Go's (1.8) garbage collector: https://twitter.com/brianhatfield/status/804355831080751104 https://twitter.com/brianhatfield/status/804355831080751104. I think it'll be alright.
hbs 10y ago
Performance usually varies greatly with your set up. I have not tried the 1.1 release but earlier relases suffered very badly when authentication was turned on and when data came out of order. Nobody cares about the ideal case, it's never encountered.
iurisilvio 10y ago
I don't have too much data (around 2 years and 500k data points. I was about to move to another database, because a query to old data or long ranges just consumed all my machine resources for some minutes. I can't use grafana because their reads crashes the server when someone want a different query. After the 1.1 upgrade, it is really fixed. I can query anything without worries.
aorth 10y ago
I wanted to try it, but the downloads page presents a popup asking for your name and email address, and you cannot dismiss it. https://www.influxdata.com/downloads/ https://www.influxdata.com/downloads/
- reacharavindh 10y ago
  And what is wrong with it? Honestly curious.
  - aorth 10y ago
    I understand the tradeoff between them giving me free software and me giving them my contact information—the way they've done it is just so up front and in your face! I'm pretty sure there is a web dev or user experience best practice about not spamming your users with "subscribe to us" and "like us on Facebook" dialogs immediately when they land on your page. And this one can't even be dismissed unless you subscribe!
    - daenney 10y ago
      This is what I usually use sites like http://10minutemail.com http://10minutemail.com for.
      - skeptic2718 10y ago
        Or in this specific case, just enter garbage in the form.
  - daenney 10y ago
    As far as I'm concerned I shouldn't have to give up my contact details in order to download FOSS software. Thankfully in this case I can just go build the thing myself but these kind of sign up walls without a way to opt out of them do annoy me.
    - reacharavindh 10y ago

121 points by otoolep 10y ago

Very good to hear that. Started to try influxdb as my tsdb for iot and looks very promising.

Currently using internally for production level work. They need to spend serious time making it easier to diagnose issues when they happen. The log is simply not verbose enough and when I hit issues with telegram, kapacitor and influxdb I have to open up a github issue having no idea what is causing the issue. Still highly recommended they just need to spend time making it easier to identify common problems. It'll help reduce the number of github issues they have opened too.

chrissnell 10y ago

I'm looking forward to trying it out. I used it at home to build a live weather site for my backyard weather station: https://mhkweather.com https://mhkweather.com The performance went way down when I moved my DB from a ext4 SSD volume to a four-disk ZFS magnetic volume. I expected some slowdown but not as much as I saw. Looking forward to trying 1.1 to see if it improves things.

otoolep 10y ago

Nice work by the InfluxDB team.

regecks 10y ago

Sweet. I had some 'getting oomkilled' problems with boundless queries back when I started using Influx for production early in the year, but it has been rock solid and performant since the later 0.1x releases. > Support for regular expressions on field keys in the SELECT clause. For example, SELECT /cpu_\d/ FROM cpu Woohoo! Very nice for making Grafana dashboards. > The admin UI (port 8083) is now officially deprecated and disabled in this release Ahh. I must be the only one using it, it gave kind of a good overview of all your collections. A bit of a pain to do from Grafana but oh well.

CSDude 10y ago
The admin UI was really good and they started to strip features away from it one by one and now it is disabled. It even could do simple graphs and was very useful for debugging.
- zenlot 10y ago
  That seems to be their uncommunicated plan - to attract users to open source version and then remove features to offer just with enterprise product. They have done it with clustering support where after attracting people with promises they then removed it from open source version. Grafana as a replacement? It really does different things.
  - omn1 10y ago
    That sounds a bit unfair to me. In the end it's a company and they need to earn money somehow. Clustering is a very hard problem that takes a lot of manpower to get right.
    - imglorp 10y ago
      They do deserve to be paid for their product, yes. And it's worth paying for. They ill will was also deserved though, because when they changed business plans to exclude more features from the community version, they surprised and angered many users. Bait and switch is what got communicated, and trust violated, forcing projects to seek alternatives or modify their budgets. This can be avoided by clearly communicating the roadmap of community vs enterprise features and sticking to it. But don't keep danging new features in free and then remove them unexpectedly.
  - pauldix 10y ago
    The admin UI functionality is getting moved over to Chronograf, which is now open source. We do have commercial stuff, but trying to make money off our UI was never a strategy.
omn1 10y ago

c2h5oh 10y ago

Tried using in production, failed really badly performance-wise (granted, it wasn't a good fit - cardinality was high). Replaced with PostgreSQL and I'm seeing 2+ orders of magnitude improvement on c4.large vs r4.8xlarge before..

scrollaway 10y ago
Sounds like you were using the wrong tool for the job. Influx is not for high cardinality data.
- hbs 10y ago
  The problem is the definition of high cardinality, the Gorilla paper talks about 2B metrics at FB but the hardware sizing page for influx shows a scary graph which basically kills any series cardinality above a million https://docs.influxdata.com/influxdb/v1.1/guides/hardware_sizing/ https://docs.influxdata.com/influxdb/v1.1/guides/hardware_si... quite a difference!
  - user5994461 10y ago
    InfluxData is a single server app with zero failover and zero sharding. There is no "billion" metrics going in there. The architecture (or lack thereof) is not intended for that. You can try vertical scaling (if you're not in the cloud), you'll squeeze more metrics until you overflow your network cards or your hard drives or your CPU.
agentgt 10y ago
In the same boat. I tried Druid as well which was almost a better solution but lacked the flexibility and maturity. We are using Postgres + citus + some HLL. Our success is probably because we just have a lot more experience with Postgres. I still haven't tried HBase though.

daenney 10y ago

Can anyone verify the claims on their production setup? I'm always a bit weary of these announcements. It's easier to get big numbers like that on benchmarks than in real usage scenarios, has usually been my experience.

siscia 10y ago

I am wondering the choice of go as language to build a database. I am a big fan of golang for some workload but the GC will kill performance in a database, wouldn't? I am not being a brat I am seriously wondering how much GC affect the overall performance.

rakoo 10y ago
Then you should be happy that Go has been putting a lot of effort into making its GC as non-blocking as possible. There was an interesting blog post from Pusher yesterday (https://blog.pusher.com/golangs-real-time-gc-in-theory-and-practice/ https://blog.pusher.com/golangs-real-time-gc-in-theory-and-p...) about how they had to learn how the GC works and how to make the most of it; the links give more information into golang's internal evolution.
maaaats 10y ago
> but the GC will kill performance in a database, wouldn't? Why do you assume it would?
daenney 10y ago
> I am a big fan of golang for some workload but the GC will kill performance in a database, wouldn't? Cassandra and HSQLDB are written in Java for example which usually suffer from much larger GC collections/pauses (though you can read a book or two on the subject and tune it). So is Neo4j, a graph database. Or Datomic in Clojure (which runs on the JVM). Then there's Mnesia written in Erlang. A time series database, or a graph database, is quite different from a more general purpose database like Postgres/MySQL and means you need to consider different things. GC isn't necessarily an issue for high throughput (read or write). A lot of it depends on how much pressure you're putting on the GC (this is true for any type of application). Being aware of how your chosen language's GC behaves paired with the (typical) usage/behaviour of your code is probably the more important thing to consider. Just because something is written in a language without a GC doesn't automatically make it performant either.
- daenney 10y ago
  Oh, and about Go's (1.8) garbage collector: https://twitter.com/brianhatfield/status/804355831080751104 https://twitter.com/brianhatfield/status/804355831080751104. I think it'll be alright.

hbs 10y ago

Performance usually varies greatly with your set up. I have not tried the 1.1 release but earlier relases suffered very badly when authentication was turned on and when data came out of order. Nobody cares about the ideal case, it's never encountered.

iurisilvio 10y ago

I don't have too much data (around 2 years and 500k data points. I was about to move to another database, because a query to old data or long ranges just consumed all my machine resources for some minutes. I can't use grafana because their reads crashes the server when someone want a different query. After the 1.1 upgrade, it is really fixed. I can query anything without worries.

aorth 10y ago

I wanted to try it, but the downloads page presents a popup asking for your name and email address, and you cannot dismiss it. https://www.influxdata.com/downloads/ https://www.influxdata.com/downloads/

reacharavindh 10y ago
And what is wrong with it? Honestly curious.
- aorth 10y ago
  I understand the tradeoff between them giving me free software and me giving them my contact information—the way they've done it is just so up front and in your face! I'm pretty sure there is a web dev or user experience best practice about not spamming your users with "subscribe to us" and "like us on Facebook" dialogs immediately when they land on your page. And this one can't even be dismissed unless you subscribe!
  - daenney 10y ago
    This is what I usually use sites like http://10minutemail.com http://10minutemail.com for.
    - skeptic2718 10y ago
      Or in this specific case, just enter garbage in the form.
- daenney 10y ago
  As far as I'm concerned I shouldn't have to give up my contact details in order to download FOSS software. Thankfully in this case I can just go build the thing myself but these kind of sign up walls without a way to opt out of them do annoy me.
  - reacharavindh 10y ago