This is a great article, and if I were writing path-critical code for the situation the author describes I'd use an AST too, but I still munge me some text and

This is a great article, and if I were writing path-critical code for the situation the author describes I'd use an AST too, but I still munge me some text and most of the time it's good enough. The problem with using AST libraries is that they're hard to understand unless you're used to them. So if you have to choose between a couple hundred lines of python or ruby and a giant AST I think it's smarter to go with the ruby. That way when something like a new emoji comes out, or what have you, the junior dev can handle the minor bug fix. Your AST is still going to choke on it. No way around it, the world has changed. That said, there is an art to writing text processors and keeping things sane. The most important bit I've learned is to avoid transformations during extractions even at the cost of performance. At least until a fully exhaustive test suite is up. It makes it easier to test and it's a clearer separation of responsibilities. I've written a lot of these parsers. One for a custom format for the Nasdaq on a project I was the lead on! I shudder to think of how many bits that thing is processing since it pipes data to and from thousands of data brokers and hedge funds.