3 ms·

Parsing is an area that I'm interested in. Can you talk more about your experience getting LLMs to one-shot parsers? From scratch LLMs seem to be completely l

by Verdex 1y ago

Parsing is an area that I'm interested in. Can you talk more about your experience getting LLMs to one-shot parsers? From scratch LLMs seem to be completely lost writing parsers. The bleeding edge appears to be able to maybe parse xml, but gives up on programming languages with even the most minimal complexity (an example being C where Gemini refused to even try with macros and then when told to parse C without macros gave an answer with several stubs where I was supposed to fill in the details). With parsing libraries they seem better, but ultimately that reduces to transform this bnf. Which if I had to I could do deterministically without an LLM. Also, my best 'successes' have been along the lines of 'parse in this well defined language that just happens to have dozens if not hundreds of verbatim examples on github'. Anytime I try to give examples of a hypothetical language then they return a bunch of regex that would not work in general.

wrs 1y ago
A few weeks ago I gave an LLM (Gemini 2.5 something in Cursor) a bunch of examples of a new language, and asked it to write a recursive descent parser in Ruby. The language was nothing crazy, intentionally reminiscent of C/JS style, but certainly the exact definition was new. I didn’t want to use a parser generator because (a) I’d have to learn a new one for Ruby, and (b) I’ve always found it easier to generate useful error messages with a handwritten recursive descent parser. IIRC, it went like this: I had it first write out the BNF based on the examples, and tweaked that a bit to match my intention. Then I had it write the lexer, and a bunch of tests for the lexer. I had it rewrite the lexer to use one big regex with named captures per token. Then I told it to write the parser. I told it to try again using a consistent style in the parser functions (when to do lookahead and how to do backtracking) and it rewrote it. I told it to write a bunch of parser tests, which I tweaked and refactored for readability (with LLM doing the grunt work). During this process it fixed most of its own bugs based on looking at failed tests. Throughout this process I had to monitor every step and fix the occasional stupidity and wrong turn, but it felt like using a power tool, you just have to keep it aimed the right way so it does what you want. The end result worked just fine, the code is quite readable and maintainable, and I’ve continued with that codebase since. That was a day of work that would have taken me more like a week without the LLM. And there is no parser generator I’m aware of that starts with examples rather than a grammar.
- Verdex 1y ago
  Thanks for giving details about your workflow. At least for me it helps a lot in these sorts of discussions. Although, it is interesting to me that the original posting mentioned LLMs "one-shot"ing parsers and this description sounds like a much more in depth process. "And there is no parser generator [...] that starts with examples [...]" People. People can generate parsers by starting with examples. Which, again, is more in line with the original "one-shot parsers" comment. If people are finding LLMs useful as part of a process for parser generation then I'm glad. (And I mean testing parsers is pretty painful to me so I'm interested in the test case generation). However I'm much more interested in the existence or non-existent of one-shot parser generation.
  - steveklabnik 1y ago
    I recently did something similar, but different: gave Claude some code examples of a Rust-like language, it wrote a recursive descent parser for me. That was a one-shot, though it's a very simple language. After more features were added, I decided I wanted BNF for it, so it went and wrote it all out correctly, after the fact, from the parser implementation.
    - Verdex 1y ago
      Can you give more info? How big of a number is "some"? Also what kind of prompts were you feeding it? Did you describe it as Rust like? Anything else you feel is relevant. [Is there a GitHub link? I'm more than happy to do the detective work.]
      - steveklabnik 1y ago
        Like three or four. very simple language: main function whos value is the error code, functions of one argument returning one value, only ints, basic control flow and math. I just opened the repo, here's the commit that did what I'm talking about: https://github.com/steveklabnik/rue/commit/5742e7921f241368e1ab7027df8d4a14e5a9bb1f#diff-d7d183da3129291ef1210410b2898ac6a65a4e4de8ede9775601173545570f57 https://github.com/steveklabnik/rue/commit/5742e7921f241368e... Well, the second part anyway, with the grammar. It writing the lexer starts as https://github.com/steveklabnik/rue/commit/a9bce389ea358365f05673576a0f6ffcf5c787da#diff-bc6661da34ecae62fbe724bb93fd69b91a7f81143f2683a81163231de7e3b545R21-R25 https://github.com/steveklabnik/rue/commit/a9bce389ea358365f..., it was basically this program. If I wrote down the prompts, I'd share them, but I didn't. Please ignore the large amount of llm bullshit in here, since it was private while I did this, I wasn't really worried about how annoying and slightly wrong the README etc was. HEAD is better in that regard.
deleted 1y ago
[deleted]

by Verdex 1y ago

wrs 1y ago

A few weeks ago I gave an LLM (Gemini 2.5 something in Cursor) a bunch of examples of a new language, and asked it to write a recursive descent parser in Ruby. The language was nothing crazy, intentionally reminiscent of C/JS style, but certainly the exact definition was new. I didn’t want to use a parser generator because (a) I’d have to learn a new one for Ruby, and (b) I’ve always found it easier to generate useful error messages with a handwritten recursive descent parser. IIRC, it went like this: I had it first write out the BNF based on the examples, and tweaked that a bit to match my intention. Then I had it write the lexer, and a bunch of tests for the lexer. I had it rewrite the lexer to use one big regex with named captures per token. Then I told it to write the parser. I told it to try again using a consistent style in the parser functions (when to do lookahead and how to do backtracking) and it rewrote it. I told it to write a bunch of parser tests, which I tweaked and refactored for readability (with LLM doing the grunt work). During this process it fixed most of its own bugs based on looking at failed tests. Throughout this process I had to monitor every step and fix the occasional stupidity and wrong turn, but it felt like using a power tool, you just have to keep it aimed the right way so it does what you want. The end result worked just fine, the code is quite readable and maintainable, and I’ve continued with that codebase since. That was a day of work that would have taken me more like a week without the LLM. And there is no parser generator I’m aware of that starts with examples rather than a grammar.

Verdex 1y ago
Thanks for giving details about your workflow. At least for me it helps a lot in these sorts of discussions. Although, it is interesting to me that the original posting mentioned LLMs "one-shot"ing parsers and this description sounds like a much more in depth process. "And there is no parser generator [...] that starts with examples [...]" People. People can generate parsers by starting with examples. Which, again, is more in line with the original "one-shot parsers" comment. If people are finding LLMs useful as part of a process for parser generation then I'm glad. (And I mean testing parsers is pretty painful to me so I'm interested in the test case generation). However I'm much more interested in the existence or non-existent of one-shot parser generation.
- steveklabnik 1y ago
  I recently did something similar, but different: gave Claude some code examples of a Rust-like language, it wrote a recursive descent parser for me. That was a one-shot, though it's a very simple language. After more features were added, I decided I wanted BNF for it, so it went and wrote it all out correctly, after the fact, from the parser implementation.
  - Verdex 1y ago
    Can you give more info? How big of a number is "some"? Also what kind of prompts were you feeding it? Did you describe it as Rust like? Anything else you feel is relevant. [Is there a GitHub link? I'm more than happy to do the detective work.]
    - steveklabnik 1y ago
      Like three or four. very simple language: main function whos value is the error code, functions of one argument returning one value, only ints, basic control flow and math. I just opened the repo, here's the commit that did what I'm talking about: https://github.com/steveklabnik/rue/commit/5742e7921f241368e1ab7027df8d4a14e5a9bb1f#diff-d7d183da3129291ef1210410b2898ac6a65a4e4de8ede9775601173545570f57 https://github.com/steveklabnik/rue/commit/5742e7921f241368e... Well, the second part anyway, with the grammar. It writing the lexer starts as https://github.com/steveklabnik/rue/commit/a9bce389ea358365f05673576a0f6ffcf5c787da#diff-bc6661da34ecae62fbe724bb93fd69b91a7f81143f2683a81163231de7e3b545R21-R25 https://github.com/steveklabnik/rue/commit/a9bce389ea358365f..., it was basically this program. If I wrote down the prompts, I'd share them, but I didn't. Please ignore the large amount of llm bullshit in here, since it was private while I did this, I wasn't really worried about how annoying and slightly wrong the README etc was. HEAD is better in that regard.

deleted 1y ago

[deleted]