5 ms·

Pyroscope and Grafana Phlare join together

187 points by buro9 4y ago

m-o11y 4y ago
This is great news. At the company I work at, OpsVerse, we offer a single-click, packaged, pre-configured OSS-driven observability stack [1] that makes heavy use of Grafana (UI + Loki) as well as Pyroscope so it's great to see them take the route of merging projects rather than building a whole new project. Continuous application profiling is such a game-changer and as eBPF becomes more popular, I'm sure tools like pyroscope will become bread and butter for devs. As an addon, we recently published a blog on how we ended us dogfooding pyroscope to debug a pesky memory leak [2]. --- [1]: https://opsverse.io/observenow-observability/ https://opsverse.io/observenow-observability/ [2]: https://opsverse.io/2023/02/09/pyroscope-to-the-rescue-debugging-memory-leak/ https://opsverse.io/2023/02/09/pyroscope-to-the-rescue-debug...
- hu3 4y ago
  Interesting! What's the performance impact of continuous profiling in production for Node.js?
  - m-o11y 4y ago
    We've heard of 2-4% performance hit in terms of performance using pyroscope on Node, but in practice, we've seen ~1% hit in terms of perf (Though we're not running high-performance systems where we're trying to eke out every bit of perf from each pod, so YMMV)
    - petethepig 4y ago
      Dmitry here (Pyroscope cofounder) This sounds about right. My favorite little anecdote that I like to tell is that the first thing people often see when they add Pyroscope to their apps is that it takes way less CPU than other signals like tracing or logging. It's pretty common to see logging taking 5-10% of overall CPU utilization. The other 90% is usually spent doing serialization / deserialization (half-joking).
netingle 4y ago
Hi! Tom from Grafana Labs here, super excited to welcome the Pyroscope team and can’t wait to see what they achieve. Continuous profiling is a next big thing IMO - easier to get started with than distributed tracing and delivers immediate value.
- abuani 4y ago
  This is exciting news Tom! Looking forward to seeing some posts in the future on how using Pyroscope reduced spend by finding perf improvements.
- anbotero 4y ago
  I’m really, really liking this. I’ve been testing the whole stack on a local server, finding kinks, documenting workflows, because I hope I move us to this soon. Now with Pyroscope I would love to try it out even more. I kid you not, we were just testing Datadog Continuous Profiling for our legacy Ruby application not two days ago and it was quite lackluster. Not saying this will be better (have yet to try), but I’d prefer to support and report feedback back on this for a brighter future for our community. Keep it up!
  - Rperry2174 4y ago
    Ryan here from Pyroscope -- Awesome to hear you're interested in migrating over. Would love for you to try out our Ruby integration and let us know what you think: https://pyroscope.io/docs/ruby/ https://pyroscope.io/docs/ruby/. We've added a lot of features like tags/labels, integration with tracing: https://github.com/pyroscope-io/otel-profiling-ruby https://github.com/pyroscope-io/otel-profiling-ruby, integration with CI/CD (in rails), etc. Feedback very welcome!
  - felixge 4y ago
    > we were just testing Datadog Continuous Profiling Felix from Datadog here :). We'd love to hear your thoughts on our profiler if you're willing to share them. My e-mail is in my profile. PS: Congrats to Ryan, Dmitry and team :).
bovermyer 4y ago
This is the last piece of the puzzle for migrating away from New Relic. Interested to see where this goes.
prasadlingawar 4y ago
This is so good, congrats Ryan.
lbrito 4y ago
I've worked with Dmitry and Ryan - both are fantastic engineers. So happy to see this. Congratulations!
Thaxll 4y ago
Curious to know how much code is kept from Phlare? How did you merged two similar projects.
- buro9 4y ago
  This is being worked out, but the gist is likely that we end up merging things rather than choosing between them. Pyroscope have driven more of the clients and have a deeper understanding of the UI — so we'll probably bias towards Pyroscope for those — and Phlare has driven more of the scalable storage and database — so we'll probably bias towards Phlare for the database. Both teams bring different perspectives of the same problem space, and the two products and their teams complement each other well, so we'll figure out how to take the best from each. The biggest win is the people, the Pyroscope people are incredible and they join a few of our best people at Grafana Labs who were already working on Phlare — in doing so they create this high talent density with a strong purpose to make profiling be an essential, easy-to-use and scalable 4th pillar of observability. For me that is the real potential, to amplify the possibilities of both teams and products by bringing them together. (I work at Grafana Labs, check my profile)
starkparker 4y ago
Pyroscope's license has already been converted from Apache to AGPL v3. Two commits on LICENSE, second was today a647634d6c05db86cd6b066d31323456528f9bf0: https://github.com/grafana/pyroscope/blob/a647634d6c05db86cd6b066d31323456528f9bf0/LICENSE https://github.com/grafana/pyroscope/blob/a647634d6c05db86cd... Previous: https://github.com/grafana/pyroscope/commit/a539a5ff69a4a39047020b06e07067e6032305ec https://github.com/grafana/pyroscope/commit/a539a5ff69a4a390...
- ericyan 4y ago
  Previous discussions when they relicensed Grafana, Loki, and Tempo to AGPL back in 2021: https://news.ycombinator.com/item?id=26877528 https://news.ycombinator.com/item?id=26877528
zokier 4y ago
I wonder if Grafana is going to rebrand at some point. They have been aggressively diversifying, but keeping the Grafana name makes people immediately associate the company with the Grafana product and consider the rest as side-projects.
- Abekkus 4y ago
  As an engineer, I care about the other products of monitoring: tickets, alerts, reports. My management? The people with the money? All they think about from that stack is the graphs they can view on Grafana.
foxinabox87 4y ago
[dead]
IYasha 4y ago
I've heard "new" Grafana is running something like chromium for server-side rendering under the hood. Is it true? (it's freaking me out a bit) Sorry if it's a silly question.
- hairyhenderson 4y ago
  What is "new" Grafana in this context? The only thing that sounds similar to this is the Grafana Image Renderer plugin, which uses Chromium. But that's hardly new (it's been around for about 4 years), and it's not part of core Grafana...

187 points by buro9 4y ago

This is great news. At the company I work at, OpsVerse, we offer a single-click, packaged, pre-configured OSS-driven observability stack [1] that makes heavy use of Grafana (UI + Loki) as well as Pyroscope so it's great to see them take the route of merging projects rather than building a whole new project. Continuous application profiling is such a game-changer and as eBPF becomes more popular, I'm sure tools like pyroscope will become bread and butter for devs. As an addon, we recently published a blog on how we ended us dogfooding pyroscope to debug a pesky memory leak [2]. --- [1]: https://opsverse.io/observenow-observability/ https://opsverse.io/observenow-observability/ [2]: https://opsverse.io/2023/02/09/pyroscope-to-the-rescue-debugging-memory-leak/ https://opsverse.io/2023/02/09/pyroscope-to-the-rescue-debug...

hu3 4y ago
Interesting! What's the performance impact of continuous profiling in production for Node.js?
- m-o11y 4y ago
  We've heard of 2-4% performance hit in terms of performance using pyroscope on Node, but in practice, we've seen ~1% hit in terms of perf (Though we're not running high-performance systems where we're trying to eke out every bit of perf from each pod, so YMMV)
  - petethepig 4y ago
    Dmitry here (Pyroscope cofounder) This sounds about right. My favorite little anecdote that I like to tell is that the first thing people often see when they add Pyroscope to their apps is that it takes way less CPU than other signals like tracing or logging. It's pretty common to see logging taking 5-10% of overall CPU utilization. The other 90% is usually spent doing serialization / deserialization (half-joking).

netingle 4y ago

Hi! Tom from Grafana Labs here, super excited to welcome the Pyroscope team and can’t wait to see what they achieve. Continuous profiling is a next big thing IMO - easier to get started with than distributed tracing and delivers immediate value.

abuani 4y ago
This is exciting news Tom! Looking forward to seeing some posts in the future on how using Pyroscope reduced spend by finding perf improvements.
anbotero 4y ago
I’m really, really liking this. I’ve been testing the whole stack on a local server, finding kinks, documenting workflows, because I hope I move us to this soon. Now with Pyroscope I would love to try it out even more. I kid you not, we were just testing Datadog Continuous Profiling for our legacy Ruby application not two days ago and it was quite lackluster. Not saying this will be better (have yet to try), but I’d prefer to support and report feedback back on this for a brighter future for our community. Keep it up!
- Rperry2174 4y ago
  Ryan here from Pyroscope -- Awesome to hear you're interested in migrating over. Would love for you to try out our Ruby integration and let us know what you think: https://pyroscope.io/docs/ruby/ https://pyroscope.io/docs/ruby/. We've added a lot of features like tags/labels, integration with tracing: https://github.com/pyroscope-io/otel-profiling-ruby https://github.com/pyroscope-io/otel-profiling-ruby, integration with CI/CD (in rails), etc. Feedback very welcome!
- felixge 4y ago
  > we were just testing Datadog Continuous Profiling Felix from Datadog here :). We'd love to hear your thoughts on our profiler if you're willing to share them. My e-mail is in my profile. PS: Congrats to Ryan, Dmitry and team :).

bovermyer 4y ago

This is the last piece of the puzzle for migrating away from New Relic. Interested to see where this goes.

prasadlingawar 4y ago

This is so good, congrats Ryan.

lbrito 4y ago

I've worked with Dmitry and Ryan - both are fantastic engineers. So happy to see this. Congratulations!

Thaxll 4y ago

Curious to know how much code is kept from Phlare? How did you merged two similar projects.

buro9 4y ago
This is being worked out, but the gist is likely that we end up merging things rather than choosing between them. Pyroscope have driven more of the clients and have a deeper understanding of the UI — so we'll probably bias towards Pyroscope for those — and Phlare has driven more of the scalable storage and database — so we'll probably bias towards Phlare for the database. Both teams bring different perspectives of the same problem space, and the two products and their teams complement each other well, so we'll figure out how to take the best from each. The biggest win is the people, the Pyroscope people are incredible and they join a few of our best people at Grafana Labs who were already working on Phlare — in doing so they create this high talent density with a strong purpose to make profiling be an essential, easy-to-use and scalable 4th pillar of observability. For me that is the real potential, to amplify the possibilities of both teams and products by bringing them together. (I work at Grafana Labs, check my profile)

starkparker 4y ago

Pyroscope's license has already been converted from Apache to AGPL v3. Two commits on LICENSE, second was today a647634d6c05db86cd6b066d31323456528f9bf0: https://github.com/grafana/pyroscope/blob/a647634d6c05db86cd6b066d31323456528f9bf0/LICENSE https://github.com/grafana/pyroscope/blob/a647634d6c05db86cd... Previous: https://github.com/grafana/pyroscope/commit/a539a5ff69a4a39047020b06e07067e6032305ec https://github.com/grafana/pyroscope/commit/a539a5ff69a4a390...

ericyan 4y ago
Previous discussions when they relicensed Grafana, Loki, and Tempo to AGPL back in 2021: https://news.ycombinator.com/item?id=26877528 https://news.ycombinator.com/item?id=26877528

zokier 4y ago

I wonder if Grafana is going to rebrand at some point. They have been aggressively diversifying, but keeping the Grafana name makes people immediately associate the company with the Grafana product and consider the rest as side-projects.

Abekkus 4y ago
As an engineer, I care about the other products of monitoring: tickets, alerts, reports. My management? The people with the money? All they think about from that stack is the graphs they can view on Grafana.

foxinabox87 4y ago

[dead]

IYasha 4y ago

I've heard "new" Grafana is running something like chromium for server-side rendering under the hood. Is it true? (it's freaking me out a bit) Sorry if it's a silly question.

hairyhenderson 4y ago
What is "new" Grafana in this context? The only thing that sounds similar to this is the Grafana Image Renderer plugin, which uses Chromium. But that's hardly new (it's been around for about 4 years), and it's not part of core Grafana...