4 ms·

I notice you are using the Go Regexp library for extracting log data. Have you noticed any performance issues with the regexps, or any bump in performance betwe

• AYBABTME 13y ago
  My understanding regarding this benchmark is that Go under-performs because the benchmarks uses a small input. For bigger inputs, Go should outperformed PCRE based Regexp, at least given this little article: http://swtch.com/~rsc/regexp/regexp1.html http://swtch.com/~rsc/regexp/regexp1.html
  • conorh 13y ago
    That article, at least in part, explains that the Go Regexp engine handles regexps that would cause pathological behavior in backtracking implementations. From the article: "Some might argue that this test is unfair to the backtracking implementations, since it focuses on an uncommon corner case. This argument misses the point: given a choice between an implementation with a predictable, consistent, fast running time on all inputs or one that usually runs quickly but can take years of CPU time (or more) on some inputs, the decision should be easy. Also, while examples as dramatic as this one rarely occur in practice, less dramatic ones do occur. Examples include using (.) (.) (.) (.) (.*) to split five space-separated fields, or using alternations where the common cases are not listed first. As a result, programmers often learn which constructs are expensive and avoid them, or they turn to so-called optimizers. Using Thompson's NFA simulation does not require such adaptation: there are no expensive regular expressions." Since the Go Regexp implementation trades off speed for predictability in performance, I'm wondering if in the real world the performance hit is noticeable.
    • voidlogic 13y ago
      It depends on if regex performance is a bottleneck in your app. I typically use the standard regexp package unless I noticed its a problem (which isn't often), then I switch to a package with the same API that uses PCRE [1]. 1: https://github.com/glenn-brown/golang-pkg-pcre https://github.com/glenn-brown/golang-pkg-pcre
• fizx 13y ago
  Regexps are slow-ish. Several MB/second/core for applying the pattern in https://github.com/onemorecloud/dendrite/blob/master/cookbook/solr.yaml https://github.com/onemorecloud/dendrite/blob/master/cookboo..., vs pcre at dozens of MB/second/core. We'll probably put pcre into Dendrite sooner or later. Dendrite is actually a port of an unreleased library I wrote in C with pcre. For us, the slowdown (which at our log velocity of hundreds of lines per second per box, still looks like 0% cpu in top), was well worth the productivity we got from moving to Go.
  • conorh 13y ago
    Thanks, that's useful information.
  • voidlogic 13y ago
    This is a near drop in replacement for "regexp" that uses PCRE: https://github.com/glenn-brown/golang-pkg-pcre https://github.com/glenn-brown/golang-pkg-pcre
    • fizx 13y ago
      I'm pretty interested, but most concerned about keeping cross-compilation intact for ease-of-install as the community grows. Sooner or later, we'll have the resources to do cross-compilation and pcre. If you want to jump on the google group or github and convince me otherwise, I'm happy to listen.