7 ms·

Show HN: Dendrite, a golang log parser

32 points by fizx 14y ago

fizx 14y ago
Author here. I wrote this to power a distributed tail of all of our logs on websolr.com. This lets us do more advanced monitoring, and we'll start exposing logs and insights from logs to users directly pretty soon. Golang was a bunch of fun for this project, and lets us ship cross-platform binaries with no dependencies, which makes our lives easier on the deploy side.
zdw 14y ago
Could someone who has run this compare it to other log tailer/shipper programs like lumberjack (https://github.com/jordansissel/lumberjack https://github.com/jordansissel/lumberjack) ?
- fizx 14y ago
  Lumberjack seems protocol-specific (https://github.com/jordansissel/lumberjack#future-protocol-discussion https://github.com/jordansissel/lumberjack#future-protocol-d...), whereas Dendrite aims to be agnostic. Most of the other log tailer/shippers are in java, so untenable for RAM-constrained environments. Dendrite also has an emphasis on structuring and re-emitting logs in e.g. json, rather than say, matching specific lines and sending email when you see them, or just forwarding all logs in their original format.
conorh 14y ago
I notice you are using the Go Regexp library for extracting log data. Have you noticed any performance issues with the regexps, or any bump in performance between 1.0 and 1.1? It appears to be one of the parts of Go that doesn't perform very well yet, for example - http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=go&lang2=java&data=u64q http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t...
- AYBABTME 14y ago
  My understanding regarding this benchmark is that Go under-performs because the benchmarks uses a small input. For bigger inputs, Go should outperformed PCRE based Regexp, at least given this little article: http://swtch.com/~rsc/regexp/regexp1.html http://swtch.com/~rsc/regexp/regexp1.html
  - conorh 14y ago
    That article, at least in part, explains that the Go Regexp engine handles regexps that would cause pathological behavior in backtracking implementations. From the article: "Some might argue that this test is unfair to the backtracking implementations, since it focuses on an uncommon corner case. This argument misses the point: given a choice between an implementation with a predictable, consistent, fast running time on all inputs or one that usually runs quickly but can take years of CPU time (or more) on some inputs, the decision should be easy. Also, while examples as dramatic as this one rarely occur in practice, less dramatic ones do occur. Examples include using (.) (.) (.) (.) (.*) to split five space-separated fields, or using alternations where the common cases are not listed first. As a result, programmers often learn which constructs are expensive and avoid them, or they turn to so-called optimizers. Using Thompson's NFA simulation does not require such adaptation: there are no expensive regular expressions." Since the Go Regexp implementation trades off speed for predictability in performance, I'm wondering if in the real world the performance hit is noticeable.
    - voidlogic 14y ago
      It depends on if regex performance is a bottleneck in your app. I typically use the standard regexp package unless I noticed its a problem (which isn't often), then I switch to a package with the same API that uses PCRE [1]. 1: https://github.com/glenn-brown/golang-pkg-pcre https://github.com/glenn-brown/golang-pkg-pcre
- fizx 14y ago
  Regexps are slow-ish. Several MB/second/core for applying the pattern in https://github.com/onemorecloud/dendrite/blob/master/cookbook/solr.yaml https://github.com/onemorecloud/dendrite/blob/master/cookboo..., vs pcre at dozens of MB/second/core. We'll probably put pcre into Dendrite sooner or later. Dendrite is actually a port of an unreleased library I wrote in C with pcre. For us, the slowdown (which at our log velocity of hundreds of lines per second per box, still looks like 0% cpu in top), was well worth the productivity we got from moving to Go.
AYBABTME 14y ago
Please don't kill me but Go is the name of the language.
- kintamanimatt 14y ago
  Golang is the colloquial name given to the language because Go is the most ridiculous and unsearchable name. Try searching HN for "Go" and you'll see the results are garbage.
  - yebyen 14y ago
    Has anyone really been far even as decided to use even go want to do look more like?
    - kintamanimatt 14y ago
      Is there something wrong with what I wrote?
      - yebyen 14y ago
        Memes that should be on the front page when you search for the word "go" on any search engine website
        kintamanimatt 14y ago
        This makes no sense.
        yebyen 14y ago
        well never mind then, if you won't even plug the phrase into Google to get a laugh, I can't help you!
daemon13 14y ago
This is an excellent start! Have some q: Why did you use yaml for configs? Yaml is/can be kind of fragile (whitespace, etc). Do you consider something more practical - nginx config formats? How to set-up log collector? For example I need to collect logs from several instances. Could not find in cookbook. How do you separate logs for different hosts/applications? How do you ship logs? UDP? TCP? TLS? How is this compared vs Logstash? Thank you!
- fizx 14y ago
  > Why did you use yaml for configs? Yaml is/can be kind of fragile (whitespace, etc). Do you consider something more practical - nginx config formats? Yaml is the most common reasonably suitable format for things that look like config files. It's familiar to more developers than nginx config, therefore we hope more people can contribute. > How to set-up log collector? For example I need to collect logs from several instances. Could not find in cookbook. Did you look at the tutorial? https://github.com/onemorecloud/dendrite/blob/master/tutorial.md https://github.com/onemorecloud/dendrite/blob/master/tutoria... > How do you separate logs for different hosts/applications? Right now, the output map has a couple automatically inserted keys for hostname, shipping timestamp, and the name of the application. This isn't terribly well-documented. > How do you ship logs? UDP? TCP? TLS? Yes ;) > How is this compared vs Logstash? Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever. I think the agnostic nature of dendrite will be a big win, as dev/ops people can persist their data to multiple stores, or swap out stores more easily. The application we wrote Dendrite for currently persists logs to three different stores.
  - daemon13 14y ago
    >> How do you ship logs? UDP? TCP? TLS? > Yes ;) ok, missed it, checked again. UDP, TCP - yes. TLS - not yet? I checked tutorial, nothing there. Since most [semi]serious projects have a least some servers, I suggest to add a separate example, smth like: 1. We have 5 servers. And we collect logs to server 6. 2. These are settings you need to use for 5 servers. 3. And these are settings you need to use for the collector [IP:Port:TCPwithTLS]. > Dendrite scrapes your existing logs Does Dendrite tail? Can Dendrite consume/scrape some existing/old log files? BTW, what was your actual max throughput? Did you use some internal queue in the design, like rsyslog is doing? > Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever. I am confused here a bit. So Dendrite is like StatsD on steroids?

32 points by fizx 14y ago

fizx 14y ago

Author here. I wrote this to power a distributed tail of all of our logs on websolr.com. This lets us do more advanced monitoring, and we'll start exposing logs and insights from logs to users directly pretty soon. Golang was a bunch of fun for this project, and lets us ship cross-platform binaries with no dependencies, which makes our lives easier on the deploy side.

zdw 14y ago

Could someone who has run this compare it to other log tailer/shipper programs like lumberjack (https://github.com/jordansissel/lumberjack https://github.com/jordansissel/lumberjack) ?

fizx 14y ago
Lumberjack seems protocol-specific (https://github.com/jordansissel/lumberjack#future-protocol-discussion https://github.com/jordansissel/lumberjack#future-protocol-d...), whereas Dendrite aims to be agnostic. Most of the other log tailer/shippers are in java, so untenable for RAM-constrained environments. Dendrite also has an emphasis on structuring and re-emitting logs in e.g. json, rather than say, matching specific lines and sending email when you see them, or just forwarding all logs in their original format.

conorh 14y ago

I notice you are using the Go Regexp library for extracting log data. Have you noticed any performance issues with the regexps, or any bump in performance between 1.0 and 1.1? It appears to be one of the parts of Go that doesn't perform very well yet, for example - http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=go&lang2=java&data=u64q http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t...

AYBABTME 14y ago
My understanding regarding this benchmark is that Go under-performs because the benchmarks uses a small input. For bigger inputs, Go should outperformed PCRE based Regexp, at least given this little article: http://swtch.com/~rsc/regexp/regexp1.html http://swtch.com/~rsc/regexp/regexp1.html
- conorh 14y ago
  That article, at least in part, explains that the Go Regexp engine handles regexps that would cause pathological behavior in backtracking implementations. From the article: "Some might argue that this test is unfair to the backtracking implementations, since it focuses on an uncommon corner case. This argument misses the point: given a choice between an implementation with a predictable, consistent, fast running time on all inputs or one that usually runs quickly but can take years of CPU time (or more) on some inputs, the decision should be easy. Also, while examples as dramatic as this one rarely occur in practice, less dramatic ones do occur. Examples include using (.) (.) (.) (.) (.*) to split five space-separated fields, or using alternations where the common cases are not listed first. As a result, programmers often learn which constructs are expensive and avoid them, or they turn to so-called optimizers. Using Thompson's NFA simulation does not require such adaptation: there are no expensive regular expressions." Since the Go Regexp implementation trades off speed for predictability in performance, I'm wondering if in the real world the performance hit is noticeable.
  - voidlogic 14y ago
    It depends on if regex performance is a bottleneck in your app. I typically use the standard regexp package unless I noticed its a problem (which isn't often), then I switch to a package with the same API that uses PCRE [1]. 1: https://github.com/glenn-brown/golang-pkg-pcre https://github.com/glenn-brown/golang-pkg-pcre
fizx 14y ago
Regexps are slow-ish. Several MB/second/core for applying the pattern in https://github.com/onemorecloud/dendrite/blob/master/cookbook/solr.yaml https://github.com/onemorecloud/dendrite/blob/master/cookboo..., vs pcre at dozens of MB/second/core. We'll probably put pcre into Dendrite sooner or later. Dendrite is actually a port of an unreleased library I wrote in C with pcre. For us, the slowdown (which at our log velocity of hundreds of lines per second per box, still looks like 0% cpu in top), was well worth the productivity we got from moving to Go.

AYBABTME 14y ago

Please don't kill me but Go is the name of the language.

kintamanimatt 14y ago
Golang is the colloquial name given to the language because Go is the most ridiculous and unsearchable name. Try searching HN for "Go" and you'll see the results are garbage.
- yebyen 14y ago
  Has anyone really been far even as decided to use even go want to do look more like?
  - kintamanimatt 14y ago
    Is there something wrong with what I wrote?
    - yebyen 14y ago
      Memes that should be on the front page when you search for the word "go" on any search engine website
      - kintamanimatt 14y ago
        This makes no sense.
        yebyen 14y ago
        well never mind then, if you won't even plug the phrase into Google to get a laugh, I can't help you!

daemon13 14y ago

This is an excellent start! Have some q: Why did you use yaml for configs? Yaml is/can be kind of fragile (whitespace, etc). Do you consider something more practical - nginx config formats? How to set-up log collector? For example I need to collect logs from several instances. Could not find in cookbook. How do you separate logs for different hosts/applications? How do you ship logs? UDP? TCP? TLS? How is this compared vs Logstash? Thank you!

fizx 14y ago
> Why did you use yaml for configs? Yaml is/can be kind of fragile (whitespace, etc). Do you consider something more practical - nginx config formats? Yaml is the most common reasonably suitable format for things that look like config files. It's familiar to more developers than nginx config, therefore we hope more people can contribute. > How to set-up log collector? For example I need to collect logs from several instances. Could not find in cookbook. Did you look at the tutorial? https://github.com/onemorecloud/dendrite/blob/master/tutorial.md https://github.com/onemorecloud/dendrite/blob/master/tutoria... > How do you separate logs for different hosts/applications? Right now, the output map has a couple automatically inserted keys for hostname, shipping timestamp, and the name of the application. This isn't terribly well-documented. > How do you ship logs? UDP? TCP? TLS? Yes ;) > How is this compared vs Logstash? Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever. I think the agnostic nature of dendrite will be a big win, as dev/ops people can persist their data to multiple stores, or swap out stores more easily. The application we wrote Dendrite for currently persists logs to three different stores.
- daemon13 14y ago
  >> How do you ship logs? UDP? TCP? TLS? > Yes ;) ok, missed it, checked again. UDP, TCP - yes. TLS - not yet? I checked tutorial, nothing there. Since most [semi]serious projects have a least some servers, I suggest to add a separate example, smth like: 1. We have 5 servers. And we collect logs to server 6. 2. These are settings you need to use for 5 servers. 3. And these are settings you need to use for the collector [IP:Port:TCPwithTLS]. > Dendrite scrapes your existing logs Does Dendrite tail? Can Dendrite consume/scrape some existing/old log files? BTW, what was your actual max throughput? Did you use some internal queue in the design, like rsyslog is doing? > Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever. I am confused here a bit. So Dendrite is like StatsD on steroids?