5 ms·

As a graphite maintainer, see my other post about problems with graphite: https://news.ycombinator.com/item?id=8908423 https://news.ycombinator.com/item?id=890

by SEJeff 12y ago

As a graphite maintainer, see my other post about problems with graphite: https://news.ycombinator.com/item?id=8908423 https://news.ycombinator.com/item?id=8908423 I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform. The more OSS metrics solutions, the better!

bbrazil 12y ago
Great! We're over in #prometheus on Freenode if there's anything we can help you with.
falcolas 12y ago
Before digging into converting the backend to Prometheus, you may want to have a look at their actual persistence method. What worries me is that they are still storing one metric per file, and are copying data from one file to the next to perform checkpoints or purges (which is admittedly an improvement over whisper's method of thrashing back and forth across a single file). They have the code to do in-place method of appending chunks, but in my admittedly brief foray through the code I could not actually find any use of it.
- SEJeff 12y ago
  The graphite backend is pluggable. It isn't converting so much as adding support for. One of my favorite backends is Cyanite which ultimately stores the data in Cassandra.
- jrv 12y ago
  Which storage/purge strategy would you recommend? (see also this thread for more discussion about the storage: https://news.ycombinator.com/item?id=8996832 https://news.ycombinator.com/item?id=8996832)
mdeeks 12y ago
A huge problem I'm having with graphite right now (which is making me look at influxdb, etc) is its inability to render graphs with lots and lots of lines. For example CPU usage across a cluster of of hundreds of machines almost always times out now. I essentially am graphing this: system.frontend-.cpu-0.cpu-used Where "frontend-" expands to 200 or so machines. I'm not entirely sure where the bottleneck is here. Would love to know if you have ideas. Is this a limitation of graphite-web itself? I have large graphite install of 20+ graphite carbon nodes running on SSDs and three additional graphite-web instances in front generating graphs. Ingesting something like 1 million metrics/min. Also I didn't realize there were still graphite maintainers (seriously. not trolling). There hasn't been a release of graphite in well over a year. I assumed it was dead by now. Any idea when we'll get a fresh release?
- SEJeff 12y ago
  We are in the final stages of the last 0.9.x release, 0.9.13. From then on, we're going to be making some more noticeable changes that break some backwards compat to make the project a lot more pleasant. Note that 0.9.13 is almost ready to be cut: https://github.com/graphite-project/graphite-web/commit/78629d4c811cf7994639dfb852b01d40af8293c3 https://github.com/graphite-project/graphite-web/commit/7862... https://github.com/graphite-project/carbon/commit/e69e1eb59aaade325d37e68d920d786f280d294a https://github.com/graphite-project/carbon/commit/e69e1eb59a... https://github.com/graphite-project/whisper/commit/19ab78ad6f4aa2c34d4bd5b84bf876ad5212bc64 https://github.com/graphite-project/whisper/commit/19ab78ad6... Anything in the master branch is what will be in 0.10.0 when we're ready to cut that. I think we'll spend some more cycles in 0.10.x focusing on non-carbon / non-whisper / non-ceres backends that should allow much better scalability. Some of these include cassandra, riak, etc. For it timing out, it is a matter of general sysadmin spleunking to figure out what is wrong. It could be IO on your carbon caches, or CPU in your render servers (where it uses cairo). I'm a HUGE fan of grafana for doing 100% of the dashboards and only using graphite-web to spit out json, or alternatively to use graphite-api. Take a look at the maxDataPoints argument to see if that will help your graphs to not timeout however.
  - mdeeks 12y ago

by SEJeff 12y ago

bbrazil 12y ago

Great! We're over in #prometheus on Freenode if there's anything we can help you with.

falcolas 12y ago

Before digging into converting the backend to Prometheus, you may want to have a look at their actual persistence method. What worries me is that they are still storing one metric per file, and are copying data from one file to the next to perform checkpoints or purges (which is admittedly an improvement over whisper's method of thrashing back and forth across a single file). They have the code to do in-place method of appending chunks, but in my admittedly brief foray through the code I could not actually find any use of it.

SEJeff 12y ago
The graphite backend is pluggable. It isn't converting so much as adding support for. One of my favorite backends is Cyanite which ultimately stores the data in Cassandra.
jrv 12y ago
Which storage/purge strategy would you recommend? (see also this thread for more discussion about the storage: https://news.ycombinator.com/item?id=8996832 https://news.ycombinator.com/item?id=8996832)

mdeeks 12y ago

A huge problem I'm having with graphite right now (which is making me look at influxdb, etc) is its inability to render graphs with lots and lots of lines. For example CPU usage across a cluster of of hundreds of machines almost always times out now. I essentially am graphing this: system.frontend-.cpu-0.cpu-used Where "frontend-" expands to 200 or so machines. I'm not entirely sure where the bottleneck is here. Would love to know if you have ideas. Is this a limitation of graphite-web itself? I have large graphite install of 20+ graphite carbon nodes running on SSDs and three additional graphite-web instances in front generating graphs. Ingesting something like 1 million metrics/min. Also I didn't realize there were still graphite maintainers (seriously. not trolling). There hasn't been a release of graphite in well over a year. I assumed it was dead by now. Any idea when we'll get a fresh release?

SEJeff 12y ago
We are in the final stages of the last 0.9.x release, 0.9.13. From then on, we're going to be making some more noticeable changes that break some backwards compat to make the project a lot more pleasant. Note that 0.9.13 is almost ready to be cut: https://github.com/graphite-project/graphite-web/commit/78629d4c811cf7994639dfb852b01d40af8293c3 https://github.com/graphite-project/graphite-web/commit/7862... https://github.com/graphite-project/carbon/commit/e69e1eb59aaade325d37e68d920d786f280d294a https://github.com/graphite-project/carbon/commit/e69e1eb59a... https://github.com/graphite-project/whisper/commit/19ab78ad6f4aa2c34d4bd5b84bf876ad5212bc64 https://github.com/graphite-project/whisper/commit/19ab78ad6... Anything in the master branch is what will be in 0.10.0 when we're ready to cut that. I think we'll spend some more cycles in 0.10.x focusing on non-carbon / non-whisper / non-ceres backends that should allow much better scalability. Some of these include cassandra, riak, etc. For it timing out, it is a matter of general sysadmin spleunking to figure out what is wrong. It could be IO on your carbon caches, or CPU in your render servers (where it uses cairo). I'm a HUGE fan of grafana for doing 100% of the dashboards and only using graphite-web to spit out json, or alternatively to use graphite-api. Take a look at the maxDataPoints argument to see if that will help your graphs to not timeout however.
- mdeeks 12y ago