10 ms·
Fuzzing a DNS parser written in Go
- artursapek 11y agoCloudFlare has a constant stream of quality articles about Go, I've really been enjoying them.
- brobinson 11y agoThey are probably pushing more traffic through Go than any other group on the planet. It's nice of them to share what they come across, and it's also good PR for them. Hell, even I want to work for them now.
- eastdakota 11y agoWe're hiring in San Francisco, London, and Singapore: https://www.cloudflare.com/join-our-team https://www.cloudflare.com/join-our-team
- outworlder 11y agoOn that note: Hypothetically speaking, what would take for a wanna-be candidate that knows Go but doesn't have relevant work experience on CloudFlare's area of expertise to get up to speed?
- brobinson 11y agoI'm in Taiwan and moving to Japan in a few months, so that's out of the question even though I've got 2.5 years of Go under my belt. :(
- jacquesm 11y agoWith the exception of Google.
- outworlder 11y agoThe possible exception of Google. I wouldn't be surprised if Cloudflare had more traffic exercising Go programs.
- jrv 11y agoPossible, yeah. The highest-throughput Go usage I'm aware of within Google is https://github.com/youtube/vitess https://github.com/youtube/vitess, Youtube's MySQL load balancer: "a fundamental component of YouTube's MySQL infrastructure, serving thousands of QPS per server". Not sure how that compares to Cloudflare's global traffic though, and what other high-throughput Go usages exist at Google.
- 0xCMP 11y agoThanks for this, it'll be nice to go through this a bit and learn how Go is used from inside Google.
- flexd 11y agoApparently dl.google.com is powered by Go. I'm guessing that get's a fair bit of traffic too. http://talks.golang.org/2013/oscon-dl.slide http://talks.golang.org/2013/oscon-dl.slide
- jamwt 11y agoDropbox's block storage system is written in go, and we're pushing terabit+ bandwidth around in it; I'd say we're in the running. :-)
- jacquesm 11y agoGah, I really should hit myself over the head, I read about that and I totally forgot to take you into account. Storage systems obviously push a ton of bits. https://blogs.dropbox.com/tech/2014/07/open-sourcing-our-go-libraries/ https://blogs.dropbox.com/tech/2014/07/open-sourcing-our-go-...
- maxmcd 11y agoThe CloudFlare blog is not written in Chinese :) https://www.google.com/trends/explore#q=golang https://www.google.com/trends/explore#q=golang
- adamzubi700 11y agohttp://googler700.blogspot.com/2015/04/blog-post.html http://googler700.blogspot.com/2015/04/blog-post.html
- amichal 11y agoIt's amusing that one of the bugs they found and fixed was a buffer reuse bug -- similar to what caused heartbleed -- will help in my 'memory safe languages are not magic bullets' rebuttals.
- hueving 11y agoSo is your rebuttal that since there are some reuse bugs, reading out of bounds is ok?
- tedunangst 11y agoHow does one go from "memory safe languages are not magic bullets" to "reading out of bounds is ok"? That's not a type safe argument.
- pcwalton 11y agoI think the point is just that "memory safety is a magic bullet" is a straw man version of the actual argument, "memory safety improves security significantly over not having it".
- tedunangst 11y agoAlas, I see the strawman version trotted out at least as frequently as the more reasonable version.
- wtracy 11y agoI've seem people believe that memory leaks can't happen in programs written in garbage-collected programs. So, yeah, the belief that garbage collection and/or bounds checking is a magic bullet really needs to be addressed.
- eru 11y agoThey are damn good bullets. Just not silver bullets. Restricting mutability would help somewhat. (Another bullet.)
- _paulc 11y agoFor dnslib [1] I use a much simpler fuzzer as part of the test suite [2] - this is nothing like as sophisticated as afl-fuzz or go-fuzz in terms of understanding and exercising code paths (just takes a DNS record and randomly mutates it - though still found a number of bugs). Does anyone know if there is anything similar to afl-fuzz/go-fuzz for python? [1] https://bitbucket.org/paulc/dnslib/ https://bitbucket.org/paulc/dnslib/ [2] https://bitbucket.org/paulc/dnslib/src/22cff0cd3e13098c001076cbb5cdcfa0a08d9c3f/fuzz.py?at=default https://bitbucket.org/paulc/dnslib/src/22cff0cd3e13098c00107...
- foote 11y agohttps://bitbucket.org/jwilk/python-afl https://bitbucket.org/jwilk/python-afl
- dkubb 11y agoI'd also recommend introducing mutation testing in addition to fuzz testing. There is some overlap between the two techniques, but they approach the problem from completely different directions. Fuzz testing randomly generates inputs and uses that to find bugs where the input is improperly handled. Mutation testing randomly changes the code to make sure the tests themselves cover all the code that is written. A good approach is to use mutation testing to make sure the existing code is fully covered by the tests. Then use fuzz testing to identify new failing test cases that need to be written. Once the implementation handles the new test cases, mutation test that code to make sure there aren't any untested code paths. I used this same approach on an Relational Algebra + SQL generator lib I wrote a while ago and the result was something that was very stable -- the only bugs were due to flaws in my own understanding of the specification, not bugs in the implementation.
- eru 11y agoI'd be interested to read a more detailed write up of the process.
- thomseddon 11y agoThere really aren't many freely available DNS servers that have seen serious production usage - I'm really excited to see RRDNS open sourced as previously hinted at, does anyone know if there is a timeline for this?
- haberman 11y agoNever listen to anyone who tells you that parsers are easy. Parsers that work on simple input are easy. Parsers that get all the edge cases and diabolical input right are stupidly difficult.
- tomjakubowski 11y ago> Parsers that work on simple input are easy. Parsers that get all the edge cases and diabolical input right are stupidly difficult. Not to mention parsers which work on a vague and totally unspecified grammar - like "real world" HTTP and HTML parsers, which have to account for misbehaving clients and servers or malformed web pages (is this what you meant by 'diabolical input'?).
- haberman 11y ago> is this what you meant by 'diabolical input' Not exactly what I meant, but that's a good point too! By diabolical input, I mean things specifically designed to stress the parser, like ("{" * 100000) in JSON.
- eru 11y agoParsers can be simple, if the language you are parsing is simple. That's why you should make your languages / protocols as `weak' as possible. Prefer a finite language over a regular language over a context free language, and avoid context sensitive languages like the plague.
- tikhonj 11y agoParsers are easy. It's the grammars that are complicated. Seriously: seemingly simple things designed and described informally, in prose, can easily be incredibly complex when you expand all their details. That says a lot about those things, a little bit about the process and mindset that went into them and virtually nothing about parsing in general.
- haberman 11y ago> Parsers are easy. It's the grammars that are complicated. What is this supposed to mean? It sounds something like "Programming is easy. It's the classes/objects that are complicated."
- jrv 11y agoA Prometheus contributor also fuzzed our query language parser using go-fuzz and amongst other things discovered a really interesting Unicode-related parsing bug: https://twitter.com/PrometheusIO/status/626187068356624384 https://twitter.com/PrometheusIO/status/626187068356624384
- Arnavion 11y agoIs there anything available to fuzz parsers written in JavaScript? (I found tunz/afl-fuzz-js and some others but they seem to be for fuzzing parsers of JavaScript.)