6 ms·

High-Performance Log Parsing in Haskell: Part One

61 points by qubitcoder 12y ago

fnord123 12y ago
""" We write a lot of Python here at Safari, in fact it’s our most widely-used language. Because it’s a very high-level language, it was likely it would satisfy my second requirement. But my experience with writing log parsers in Python before–even when using tricks like lazy evaluation–led me to suspect that it would not do so well in satisfying the first requirement. """ How can one say one parse is high performance and another isn't if no comparison is made? e.g. the author could use pylogsparser and benchmark it: https://pypi.python.org/pypi/pylogsparser/0.8 https://pypi.python.org/pypi/pylogsparser/0.8 Then we can get an idea of whether the project set out to meet its goal of being quicker than python. In any event, I'd be interested to see how luajit-lpeg compares to their haskell impl. It even has a nice online test tool: http://lpeg.trink.com/share/syslog http://lpeg.trink.com/share/syslog
- mazelife 12y ago
  Author here. Thanks for pointing me to pylogsparser, I'll definitely take a look at that. Your point is well taken: without building a parallel implementation in Python, we don't have a way of knowing for sure if Haskell is faster. The only data points I have are that we've built a couple of logparsers for custom formats in Python before this, and the number of lines parsed/second was far smaller than the attoparsec-based parser.[1] It's not apples-to-apples, since the formats differ a bit, but I don't think that it has no predictive value. So in the second part of this post, which I'm working on now, I'm hoping to be able to provide a fully-functional NCSA combined log format parser in Haskell alongside the blog post. I think that would be fairly easy to benchmark since it's a common-enough log format. [1] that's just measuring the time to parse log files into some sort of structured data, not necessarily to do anything with it
  - mijoharas 12y ago
    Thanks for the article, I think you're missing some code, it seems that every time you have a `do` block in your code samples most of the code is cut off (I assume).
    - mazelife 12y ago
      Thanks for pointing that out; I think the syntax highlighter may have clobbered some code. It's now fixed.
slashnull 12y ago
Hey, that's pretty cool! After playing with Haskell for the last two years, I figured that it would be pretty damn hard to write conventional LAMP/JS webpages in Haskell, but I still wanted to try to solve some of my real world problems with it. So I'm parsing the log files that my LAMP stuff produces. So far it's a very fun and frustrating waste of time, but given my recent progress, it will probably become a slightly less fun and frustrating time saver. I just recently cleared up some confusion and type errors coming from ByteString/Lazy ByteString/Text/Whatever conversion issues (which made up 95% of the recent frustration, as every other language I'm using right now has one single sort of string (which more than often serves to contain ints ;)) ... and database CRUD and JSON typeclass instances have appeared as if by magic. Awesome! And now I can celebrate by unleashing that stuff tens of megs of logs, opening top, and then seeing the executable instantly eat up all my RAM, before slowly conceding space to the postgres server. Hopefully your post will help me implement streaming. Cheers!
- codygman 12y ago
  > After playing with Haskell for the last two years, I figured that it would be pretty damn hard to write conventional LAMP/JS webpages in Haskell, but I still wanted to try to solve some of my real world problems with it. Really? You should check out: http://adit.io/posts/2013-04-15-making-a-website-with-haskell.html http://adit.io/posts/2013-04-15-making-a-website-with-haskel...
  - slashnull 12y ago
    I already got a few libraries and frameworks up and running (including scotty, actually, which is really cute), but somehow I can never have everything I want at the same time... Some tutorials will have a few routes and a DB, some other will have an intricate rendering and templating architecture and basically no data storage solution, some are more or less a hello world with a session manager... And there's Yesod, which seems to do everything out of the box, but which has such a colossal amount of dependencies that I never got anything to build beyond fresh yesod-init setups, which still manage to fail on my current setup, despite using Stackage and sandboxes. Among what I've seen in Haskell, ORMs and web frameworks are the two types of libraries that carry the largest and most cumbersome monad transformer stacks around; making a website involves having to typecheck a huge monad transformer into another huge monad transformer. This is not easy. Not to mention that, in the world of open-source web developement, nearly all the documentation, expertise, copy-pastable code examples and standard practices are in dynamic OOP languages. Perhaps the ease of RoR, PHP and Node has let everybody forget how much stuff is involved in making modern webpages : )
dllthomas 12y ago
"In type-system theory this is called a "Sum Type" because we can define all possible representations" It is indeed a sum type, but I don't think that is why (or true of all sum types, or less true of product types). It sounds like they are conflating "enum"? As an example of a product type for whom we can define every possible representation: ((),()) has precisely one representation. As an exampe of a sum type for whom there are infinite representations: (Either String Integer) My understanding has it that it is called a sum type because 1) for finite types the number of representations is the sum of the number of representations under each tag, and 2) it behaves more generally like a sum (a^x * a^y = a^(x+y) is an equality in arithmetic, (x -> a, y -> a) is isomorphic to ((Either x y) -> a) in programming).
- mazelife 12y ago
  That's a good point. A sum type/disjoint union (by my understanding) means the type is in the union of subsets and that the subsets are pairwise disjoint, but it doesn't say that the subsets have to be finite. Since this post is really aimed at Haskell beginners, I was trying to avoid going down the rabbit hole as regards type system or set theory and I might have oversimplified in the process.
sriku 12y ago
> Sequencing parsers applicatively allows the compiler to perform static analysis on a parser without running it. This knowledge can be used to avoid things like backtracking that may slow your parser down. This is not possible when sequencing parsers monadically because the grammar of each parser depends on the previous one. However, performance results in this case are probably negligible; don’t hesitate to choose do-notation if you find it easier to read. Would be good to talk more about this and quantify too. Very interesting topic.
- codygman 12y ago
  Maybe this paper[0] on using Applicatives for performance and concurrency in Haskell's Haxl will help. Also note that there is a feature proposal[1] for Applicative Do notation. 0: http://community.haskell.org/~simonmar/papers/haxl-icfp14.pdf http://community.haskell.org/~simonmar/papers/haxl-icfp14.pd... 1: https://ghc.haskell.org/trac/ghc/wiki/ApplicativeDo https://ghc.haskell.org/trac/ghc/wiki/ApplicativeDo

61 points by qubitcoder 12y ago

""" We write a lot of Python here at Safari, in fact it’s our most widely-used language. Because it’s a very high-level language, it was likely it would satisfy my second requirement. But my experience with writing log parsers in Python before–even when using tricks like lazy evaluation–led me to suspect that it would not do so well in satisfying the first requirement. """ How can one say one parse is high performance and another isn't if no comparison is made? e.g. the author could use pylogsparser and benchmark it: https://pypi.python.org/pypi/pylogsparser/0.8 https://pypi.python.org/pypi/pylogsparser/0.8 Then we can get an idea of whether the project set out to meet its goal of being quicker than python. In any event, I'd be interested to see how luajit-lpeg compares to their haskell impl. It even has a nice online test tool: http://lpeg.trink.com/share/syslog http://lpeg.trink.com/share/syslog

mazelife 12y ago
Author here. Thanks for pointing me to pylogsparser, I'll definitely take a look at that. Your point is well taken: without building a parallel implementation in Python, we don't have a way of knowing for sure if Haskell is faster. The only data points I have are that we've built a couple of logparsers for custom formats in Python before this, and the number of lines parsed/second was far smaller than the attoparsec-based parser.[1] It's not apples-to-apples, since the formats differ a bit, but I don't think that it has no predictive value. So in the second part of this post, which I'm working on now, I'm hoping to be able to provide a fully-functional NCSA combined log format parser in Haskell alongside the blog post. I think that would be fairly easy to benchmark since it's a common-enough log format. [1] that's just measuring the time to parse log files into some sort of structured data, not necessarily to do anything with it
- mijoharas 12y ago
  Thanks for the article, I think you're missing some code, it seems that every time you have a `do` block in your code samples most of the code is cut off (I assume).
  - mazelife 12y ago
    Thanks for pointing that out; I think the syntax highlighter may have clobbered some code. It's now fixed.

slashnull 12y ago

Hey, that's pretty cool! After playing with Haskell for the last two years, I figured that it would be pretty damn hard to write conventional LAMP/JS webpages in Haskell, but I still wanted to try to solve some of my real world problems with it. So I'm parsing the log files that my LAMP stuff produces. So far it's a very fun and frustrating waste of time, but given my recent progress, it will probably become a slightly less fun and frustrating time saver. I just recently cleared up some confusion and type errors coming from ByteString/Lazy ByteString/Text/Whatever conversion issues (which made up 95% of the recent frustration, as every other language I'm using right now has one single sort of string (which more than often serves to contain ints ;)) ... and database CRUD and JSON typeclass instances have appeared as if by magic. Awesome! And now I can celebrate by unleashing that stuff tens of megs of logs, opening top, and then seeing the executable instantly eat up all my RAM, before slowly conceding space to the postgres server. Hopefully your post will help me implement streaming. Cheers!

codygman 12y ago
> After playing with Haskell for the last two years, I figured that it would be pretty damn hard to write conventional LAMP/JS webpages in Haskell, but I still wanted to try to solve some of my real world problems with it. Really? You should check out: http://adit.io/posts/2013-04-15-making-a-website-with-haskell.html http://adit.io/posts/2013-04-15-making-a-website-with-haskel...
- slashnull 12y ago
  I already got a few libraries and frameworks up and running (including scotty, actually, which is really cute), but somehow I can never have everything I want at the same time... Some tutorials will have a few routes and a DB, some other will have an intricate rendering and templating architecture and basically no data storage solution, some are more or less a hello world with a session manager... And there's Yesod, which seems to do everything out of the box, but which has such a colossal amount of dependencies that I never got anything to build beyond fresh yesod-init setups, which still manage to fail on my current setup, despite using Stackage and sandboxes. Among what I've seen in Haskell, ORMs and web frameworks are the two types of libraries that carry the largest and most cumbersome monad transformer stacks around; making a website involves having to typecheck a huge monad transformer into another huge monad transformer. This is not easy. Not to mention that, in the world of open-source web developement, nearly all the documentation, expertise, copy-pastable code examples and standard practices are in dynamic OOP languages. Perhaps the ease of RoR, PHP and Node has let everybody forget how much stuff is involved in making modern webpages : )

dllthomas 12y ago

"In type-system theory this is called a "Sum Type" because we can define all possible representations" It is indeed a sum type, but I don't think that is why (or true of all sum types, or less true of product types). It sounds like they are conflating "enum"? As an example of a product type for whom we can define every possible representation: ((),()) has precisely one representation. As an exampe of a sum type for whom there are infinite representations: (Either String Integer) My understanding has it that it is called a sum type because 1) for finite types the number of representations is the sum of the number of representations under each tag, and 2) it behaves more generally like a sum (a^x * a^y = a^(x+y) is an equality in arithmetic, (x -> a, y -> a) is isomorphic to ((Either x y) -> a) in programming).

mazelife 12y ago
That's a good point. A sum type/disjoint union (by my understanding) means the type is in the union of subsets and that the subsets are pairwise disjoint, but it doesn't say that the subsets have to be finite. Since this post is really aimed at Haskell beginners, I was trying to avoid going down the rabbit hole as regards type system or set theory and I might have oversimplified in the process.

sriku 12y ago

> Sequencing parsers applicatively allows the compiler to perform static analysis on a parser without running it. This knowledge can be used to avoid things like backtracking that may slow your parser down. This is not possible when sequencing parsers monadically because the grammar of each parser depends on the previous one. However, performance results in this case are probably negligible; don’t hesitate to choose do-notation if you find it easier to read. Would be good to talk more about this and quantify too. Very interesting topic.

codygman 12y ago
Maybe this paper[0] on using Applicatives for performance and concurrency in Haskell's Haxl will help. Also note that there is a feature proposal[1] for Applicative Do notation. 0: http://community.haskell.org/~simonmar/papers/haxl-icfp14.pdf http://community.haskell.org/~simonmar/papers/haxl-icfp14.pd... 1: https://ghc.haskell.org/trac/ghc/wiki/ApplicativeDo https://ghc.haskell.org/trac/ghc/wiki/ApplicativeDo