3 ms·
Another example. I chatted with it for quite some time to define a toy language. I asked it to write small programs like an echo server in this toy language, an
by StewardMcOy 2y ago
Another example. I chatted with it for quite some time to define a toy language. I asked it to write small programs like an echo server in this toy language, and it did pretty well (though it sometimes hallucinated changes to the language). I then asked for a tree-sitter grammar, and it always got close, but never could fix the problems. Even after I pointed out the problem, its fixes were always wrong. So instead I decided to try a different approach.
> Let's write a compiler in C that tokenizes a single file in this language, parses it into a syntax tree, and emits an object file in LLVM IR. I'd like the compiler to be readable and understandable by a human, but execute as fast as possible.
The tokenizer it wrote was good, but when it got to the operators, it implemented + and -, and then contained this comment:
// Add other operators and delimiters
When I asked it to fill in the operators, it actually did a good job. However, this was the parser it gave me.
typedef struct ASTNode {
TokenType type;
char* value;
struct ASTNode* left;
struct ASTNode* right;
} ASTNode;
ASTNode* parse(Token* tokens) {
// Example parsing logic, build your AST based on the tokens
return NULL; // Placeholder
}
Obviously, it's leaving building the entire AST up to me. So I asked it:
> Now implement the complete parser.
The result was long, so I won't paste it here, but it contained all kinds of comments:
// Skipping parameter parsing for simplicity
// parameters would go here
// Simple implementation: only supports return statements for now
// Simple expression parsing: only literals for now
// Add more parsing functions as needed...
So I said:
> Seriously, I want you to implement the entire thing, not an outline or a framework.
And it gave be a code with fewer, but more complete parsing functions. The expression function still had this comment.
// Assume we're only parsing integers and binary +,- operations for simplicity
And then at the bottom:
// Assume the rest of the necessary parsing functions are implemented similarly
- simonw 2y agoThat's really interesting, thanks. I wonder if the initial chatting puts it more in the mood to coach you rather than write the code? My prompting style is much more direct - things like this: https://chatgpt.com/share/61cd85f6-7002-4676-b204-0349a723232a?oai-dm=1 https://chatgpt.com/share/61cd85f6-7002-4676-b204-0349a72323... - more here: https://simonwillison.net/series/using-llms/ https://simonwillison.net/series/using-llms/
- StewardMcOy 2y agoInteresting. Most of the time the sentences in your prompts don't even contain a predicate. I tried that with the two examples from before. Mixed results. Still had one comment telling me to finish it myself in the SNES PPU example, but that's a lot fewer than before. Unfortunately, when I told it to do the work, it gave me a more complete, but not totally complete header without mentioning that it was incomplete. It also took a few tries and I had to regenerate a response that started generating garbage code and then error'd out. I checked to make sure that ChatGPT had no memory of previous conversations related to the SNES PPU. So I suspect this has more to do with the task given than the prompting style. https://chat.openai.com/share/8703a48a-af72-4a9f-976e-688a26724de6 https://chat.openai.com/share/8703a48a-af72-4a9f-976e-688a26... The parser was more interesting. I didn't remove the details of our toy language discussion from ChatGPT's memory, but I also didn't specify much in the way of what I wanted. It created some funky parser code, and I didn't try to compile it to see if it was correct, but it didn't appear to display the lazy behavior I saw before. It also titled the conversation in Spanish, even though I've never chatted in Spanish with it and I'm not fluent. I suspect I could have gotten more correct output by giving a more detailed description of what I wanted, but I was trying to keep it simple to match your prompting style. https://chat.openai.com/share/ccc23d98-6d4f-4be5-b9b6-b4464ff1bf9d https://chat.openai.com/share/ccc23d98-6d4f-4be5-b9b6-b4464f...