6 ms·

Stripe’s Veneur: A distributed, fault-tolerant pipeline for observability data

109 points by federicoponzi 8y ago

madspindel 8y ago
It's from 2016: https://stripe.com/blog/introducing-veneur-high-performance-and-global-aggregation-for-datadog https://stripe.com/blog/introducing-veneur-high-performance-...
- mhluongo 8y ago
  Last commit on master was 5 days ago, and appears otherwise to be maintained. A lot can change in a repo in 2 years
- chimeracoder 8y ago
  > It's from 2016 2016 was the first public release, but the project has grown a lot in that time. You can take a look at the changelog to see what's new, ever since we switched to a six-week release cycle last year: https://github.com/stripe/veneur/blob/master/CHANGELOG.md https://github.com/stripe/veneur/blob/master/CHANGELOG.md Source: I work on the Observability team at Stripe and I am the PM for Veneur.
  - martinpinto 8y ago
    yep, I can confirm. This is a very stable project!
dswalter 8y ago
It always makes me happy to see approximate algorithms/data structures like hyperloglog being used.
- ambicapter 8y ago
  "probabilistic" is probably the word you're looking for, and yes I agree, I'm fascinated by the idea of trading off a little bit of accuracy for massive performance gains.
  - dswalter 8y ago
    That's a good term. I'm sure you've found this collection of links on 'streaming algorithms'. It's a gold mine of resources in this space: https://gist.github.com/debasishg/8172796 https://gist.github.com/debasishg/8172796
pinko 8y ago
Know of anyone using this in production outside Stripe?
- galaktor 8y ago
  Intercom use it heavily in production. Source: am engineer at Intercom.
  - chimeracoder 8y ago
    > Intercom use it heavily in production. Source: am engineer at Intercom. This is awesome to hear! Are you referring to this Intercom? https://www.intercom.com/ https://www.intercom.com/
    - galaktor 8y ago
      Yes, that's the one.
- chimeracoder 8y ago
  > Know of anyone using this in production outside Stripe? In addition to Intercom (mentioned in a sibling comment), some other companies off the top of my head who are also using Veneur: Sentry, Bluecore, Axiom, Quantcast, and at least two more that I'm not sure if I have permission to name publicly. There are more too - since Veneur is an OSS project, we only find out when people submit submit PRs to the project or happen to contact us for another reason. It makes for a pleasant surprise when we find out!
- otterley 8y ago
  We use it at Segment as a replacement for Datadog's Python-based (and dog slow) dogstatsd server. Veneur is not only faster on a per-CPU basis, but scales to multiple CPUs as well. I consider it absolutely essential for collecting metrics on large instances that run hundreds of busy tasks emitting hundreds or thousands of metrics per second.
- deleted 8y ago
  [deleted]
chimeracoder 8y ago
This was a pleasant surprise to see on Hacker News this morning! I work on the Observability team at Stripe and have been the PM for Veneur (and the rest of our metrics & tracing pipeline work) pretty much since we released it ~2 years ago. If you're interested in learning more about how Veneur works and why we built it, I gave a talk at Monitorama last year that explains the philosophy behind Veneur[0]. In short, a massive company like Google is able to build their on integrated observability stacks in-house, but almost any other smaller company is going to be relying on an array of open-source tools or third-party vendors for different parts of their observability tooling[1]. When using different tools, there are always going to be gaps between them, which leads to incomplete instrumentation and awkward (inter-)operability. By taking control of the pipeline that processes the data, we're able to provide fully integrated views into different aspects of our observability data. The Monitorama talk is a year old at this point, so it doesn't cover some of the newer things Veneur has helped us to accomplish, but the core philosophy hasn't changed. I've given updated versions of the talk more recently at CraftConf (in May) and DevOpsDaysMSP (last week), but neither of those videos are online yet. [0] https://vimeo.com/221049715 https://vimeo.com/221049715 [1] e.g. ELK/Papertrail/Splunk for logs, Graphite/Datadog/SignalFx for metrics, and maybe a third tool for tracing if you're lucky.
tchaffee 8y ago
Am I the only one who is always slightly disappointed that neither the README file on Github nor the landing page at the website tells me why I would want to use the software in question? What problem it solves? Why might "a distributed, fault-tolerant observability pipeline" be interesting to programmers or anyone else? It seems like you've already got to be familiar with the problem space to understand what this is and what need it fulfills. I'm not picking on this package. I see it all the time. Can someone here explain to me what the use case is for this software?
- jmillikin 8y ago
  The use case: you have more than a hundred machines emitting lots of monitoring data, much of which is uninteresting except in aggregate form. Instead of paying to store millions of data points and then computing the aggregates later, Veneur can calculate things like percentiles itself and only forwards those. It also has a separate, related purpose as a statsd protocol transport. You run it on the statsd port and it receives standard (or DataDog-extended) UDP traffic, then it forwards metrics via TCP to another backend. This has reliability benefits when operating over a network that might drop UDP packets, such as the public internet.
  - tchaffee 8y ago
    Thanks. That's pretty good. Every README needs to start with something like this: "If you monitor applications, Veneur can save you money and increase reliability"
    - mrkurt 8y ago
      That sounds like a startup's elevator pitch, not a README for an OSS project.
      - tchaffee 8y ago
        It kind of does sound like the first line of an elevator pitch, doesn't it? And as a programmer it helps me quickly decide if I should be care about it at all, and how to categorize it. Take a look at the software this "pipeline" is supposed to integrate with. It's a commercial product. Their landing page could not be clearer: "Modern monitoring & analytics. See inside any stack, any app, at any scale, anywhere" Perfect. Now I know what they do in plain English, how to mentally categorize their product, and even how to bookmark the page for the future when I might need their product. React is an OSS product, and it comes close "A declarative, efficient, and flexible JavaScript library for building user interfaces." Tells me an awful lot with only one jargon word: declarative. But I can still categorize it mentally and bookmark it even if I don't know what that words means. Make it easy for people to understand what you do, and you'll get more interest. People will bother to bookmark it. And maybe come back.
roskilli 8y ago
It’s definitely interesting to see the different systems being built for monitoring across the different tech co’s. M3 aggregator, Uber’s metrics aggregation tier is similar, except it has inbuilt replication and leader election on top of etcd to avoid any SPOF during deployments, failed instances, etc. Also it uses Cormode-Muthukrishnan for estimating percentiles by default, it has support for T-Digest too. Although these days submitting histogram bucket aggregates all the way from the client to aggregator then to storage is more popular as you can estimate percentiles across more dimensions and time windows at query time quite cheaply. You need to choose your buckets carefully though. It too is open source, but needs some help to make it plug into other stacks more easily: https://github.com/m3db/m3aggregator https://github.com/m3db/m3aggregator
amelius 8y ago
What do they mean by "observability data"? Is this a fancy way of saying "privacy-sensitive user data"?
- Jarred 8y ago
  More like diagnostic stats about servers - memory usage, CPU usage and many more things like that.
noncoml 8y ago
Why is Go so popular in the industry at the moment? What's the decision process for choosing Go?
- ebikelaw 8y ago
  I doubt anyone can answer this for you, but why shouldn't it be? It is a very sensible language and toolchain. When writing source, it is easy to write tests and testable code and to run the tests as part of your build. At build time, it's fast and it produces easy-to-deploy statically-linked applications. At runtime, it's pretty fast and compact ... compared to python, Go is most of the way to C++-level performance.
  - noncoml 8y ago
    > why shouldn't it be? I didn't imply that it shouldn't be. I am genuinely interested to know the thought process and why it seems to be the defacto language these days. Your arguments are strong, thanks.
  - dozzie 8y ago
    > I doubt anyone can answer this for you, but why shouldn't it be [popular in the industry at the moment]? Its build system is poor if you want to rebuild the binary from the same sources (you can't precompile the libraries used), and the statically linked binary may be nice for a one-off deployment ("fire and forget" mode), but for repeated deployments and multiple versions running at the same time it reminds me the Chinese torture "death by a thousand cuts" in what you can't do with such a binary (dozens of small things that are hard to remember, each on its own not being enough to go away from static linking, but boy, they do add up).
    - ebikelaw 8y ago
      Can you remember even one? I'm interested. I've used Go at Extremely Large Scale (tm) and never thought it was terribly troublesome.
      - 8y ago
ebikelaw 8y ago
When I'm evaluating a system like this what I want to read about is how is it hardened against client stupidity. For example, someone deploys an application in my datacenter and it emits metrics that have gibberish in their names (consider a common Java bug where a class lacks a toString, so the metric gets barfed out as foo.bar.0xCAFEBABE.baz). How does the system cope with this enormous, hyper-dimensional input?

109 points by federicoponzi 8y ago

It's from 2016: https://stripe.com/blog/introducing-veneur-high-performance-and-global-aggregation-for-datadog https://stripe.com/blog/introducing-veneur-high-performance-...

mhluongo 8y ago
Last commit on master was 5 days ago, and appears otherwise to be maintained. A lot can change in a repo in 2 years
chimeracoder 8y ago
> It's from 2016 2016 was the first public release, but the project has grown a lot in that time. You can take a look at the changelog to see what's new, ever since we switched to a six-week release cycle last year: https://github.com/stripe/veneur/blob/master/CHANGELOG.md https://github.com/stripe/veneur/blob/master/CHANGELOG.md Source: I work on the Observability team at Stripe and I am the PM for Veneur.
- martinpinto 8y ago
  yep, I can confirm. This is a very stable project!

dswalter 8y ago

It always makes me happy to see approximate algorithms/data structures like hyperloglog being used.

ambicapter 8y ago
"probabilistic" is probably the word you're looking for, and yes I agree, I'm fascinated by the idea of trading off a little bit of accuracy for massive performance gains.
- dswalter 8y ago
  That's a good term. I'm sure you've found this collection of links on 'streaming algorithms'. It's a gold mine of resources in this space: https://gist.github.com/debasishg/8172796 https://gist.github.com/debasishg/8172796

pinko 8y ago

Know of anyone using this in production outside Stripe?

galaktor 8y ago
Intercom use it heavily in production. Source: am engineer at Intercom.
- chimeracoder 8y ago
  > Intercom use it heavily in production. Source: am engineer at Intercom. This is awesome to hear! Are you referring to this Intercom? https://www.intercom.com/ https://www.intercom.com/
  - galaktor 8y ago
    Yes, that's the one.
chimeracoder 8y ago
> Know of anyone using this in production outside Stripe? In addition to Intercom (mentioned in a sibling comment), some other companies off the top of my head who are also using Veneur: Sentry, Bluecore, Axiom, Quantcast, and at least two more that I'm not sure if I have permission to name publicly. There are more too - since Veneur is an OSS project, we only find out when people submit submit PRs to the project or happen to contact us for another reason. It makes for a pleasant surprise when we find out!
otterley 8y ago
We use it at Segment as a replacement for Datadog's Python-based (and dog slow) dogstatsd server. Veneur is not only faster on a per-CPU basis, but scales to multiple CPUs as well. I consider it absolutely essential for collecting metrics on large instances that run hundreds of busy tasks emitting hundreds or thousands of metrics per second.
deleted 8y ago
[deleted]

chimeracoder 8y ago

This was a pleasant surprise to see on Hacker News this morning! I work on the Observability team at Stripe and have been the PM for Veneur (and the rest of our metrics & tracing pipeline work) pretty much since we released it ~2 years ago. If you're interested in learning more about how Veneur works and why we built it, I gave a talk at Monitorama last year that explains the philosophy behind Veneur[0]. In short, a massive company like Google is able to build their on integrated observability stacks in-house, but almost any other smaller company is going to be relying on an array of open-source tools or third-party vendors for different parts of their observability tooling[1]. When using different tools, there are always going to be gaps between them, which leads to incomplete instrumentation and awkward (inter-)operability. By taking control of the pipeline that processes the data, we're able to provide fully integrated views into different aspects of our observability data. The Monitorama talk is a year old at this point, so it doesn't cover some of the newer things Veneur has helped us to accomplish, but the core philosophy hasn't changed. I've given updated versions of the talk more recently at CraftConf (in May) and DevOpsDaysMSP (last week), but neither of those videos are online yet. [0] https://vimeo.com/221049715 https://vimeo.com/221049715 [1] e.g. ELK/Papertrail/Splunk for logs, Graphite/Datadog/SignalFx for metrics, and maybe a third tool for tracing if you're lucky.

tchaffee 8y ago

Am I the only one who is always slightly disappointed that neither the README file on Github nor the landing page at the website tells me why I would want to use the software in question? What problem it solves? Why might "a distributed, fault-tolerant observability pipeline" be interesting to programmers or anyone else? It seems like you've already got to be familiar with the problem space to understand what this is and what need it fulfills. I'm not picking on this package. I see it all the time. Can someone here explain to me what the use case is for this software?

jmillikin 8y ago
The use case: you have more than a hundred machines emitting lots of monitoring data, much of which is uninteresting except in aggregate form. Instead of paying to store millions of data points and then computing the aggregates later, Veneur can calculate things like percentiles itself and only forwards those. It also has a separate, related purpose as a statsd protocol transport. You run it on the statsd port and it receives standard (or DataDog-extended) UDP traffic, then it forwards metrics via TCP to another backend. This has reliability benefits when operating over a network that might drop UDP packets, such as the public internet.
- tchaffee 8y ago
  Thanks. That's pretty good. Every README needs to start with something like this: "If you monitor applications, Veneur can save you money and increase reliability"
  - mrkurt 8y ago
    That sounds like a startup's elevator pitch, not a README for an OSS project.
    - tchaffee 8y ago
      It kind of does sound like the first line of an elevator pitch, doesn't it? And as a programmer it helps me quickly decide if I should be care about it at all, and how to categorize it. Take a look at the software this "pipeline" is supposed to integrate with. It's a commercial product. Their landing page could not be clearer: "Modern monitoring & analytics. See inside any stack, any app, at any scale, anywhere" Perfect. Now I know what they do in plain English, how to mentally categorize their product, and even how to bookmark the page for the future when I might need their product. React is an OSS product, and it comes close "A declarative, efficient, and flexible JavaScript library for building user interfaces." Tells me an awful lot with only one jargon word: declarative. But I can still categorize it mentally and bookmark it even if I don't know what that words means. Make it easy for people to understand what you do, and you'll get more interest. People will bother to bookmark it. And maybe come back.

roskilli 8y ago

It’s definitely interesting to see the different systems being built for monitoring across the different tech co’s. M3 aggregator, Uber’s metrics aggregation tier is similar, except it has inbuilt replication and leader election on top of etcd to avoid any SPOF during deployments, failed instances, etc. Also it uses Cormode-Muthukrishnan for estimating percentiles by default, it has support for T-Digest too. Although these days submitting histogram bucket aggregates all the way from the client to aggregator then to storage is more popular as you can estimate percentiles across more dimensions and time windows at query time quite cheaply. You need to choose your buckets carefully though. It too is open source, but needs some help to make it plug into other stacks more easily: https://github.com/m3db/m3aggregator https://github.com/m3db/m3aggregator

amelius 8y ago

What do they mean by "observability data"? Is this a fancy way of saying "privacy-sensitive user data"?

Jarred 8y ago
More like diagnostic stats about servers - memory usage, CPU usage and many more things like that.

noncoml 8y ago

Why is Go so popular in the industry at the moment? What's the decision process for choosing Go?

ebikelaw 8y ago
I doubt anyone can answer this for you, but why shouldn't it be? It is a very sensible language and toolchain. When writing source, it is easy to write tests and testable code and to run the tests as part of your build. At build time, it's fast and it produces easy-to-deploy statically-linked applications. At runtime, it's pretty fast and compact ... compared to python, Go is most of the way to C++-level performance.
- noncoml 8y ago
  > why shouldn't it be? I didn't imply that it shouldn't be. I am genuinely interested to know the thought process and why it seems to be the defacto language these days. Your arguments are strong, thanks.
- dozzie 8y ago
  > I doubt anyone can answer this for you, but why shouldn't it be [popular in the industry at the moment]? Its build system is poor if you want to rebuild the binary from the same sources (you can't precompile the libraries used), and the statically linked binary may be nice for a one-off deployment ("fire and forget" mode), but for repeated deployments and multiple versions running at the same time it reminds me the Chinese torture "death by a thousand cuts" in what you can't do with such a binary (dozens of small things that are hard to remember, each on its own not being enough to go away from static linking, but boy, they do add up).
  - ebikelaw 8y ago
    Can you remember even one? I'm interested. I've used Go at Extremely Large Scale (tm) and never thought it was terribly troublesome.
    - 8y ago

ebikelaw 8y ago

When I'm evaluating a system like this what I want to read about is how is it hardened against client stupidity. For example, someone deploys an application in my datacenter and it emits metrics that have gibberish in their names (consider a common Java bug where a class lacks a toString, so the metric gets barfed out as foo.bar.0xCAFEBABE.baz). How does the system cope with this enormous, hyper-dimensional input?