3 ms·

Is it a good idea to do multiple passes when asking for code? What sort of progression do we see in a multipass (!) workflow? Does anyone have any examples? (

Is it a good idea to do multiple passes when asking for code? What sort of progression do we see in a multipass (!) workflow? Does anyone have any examples? (Wrangling AIs is a skill, I think, and thus learning more is always a good idea)

jakderrida 3y ago
> Is it a good idea to do multiple passes when asking for code? Probably. However, I see their point when it comes to articles; having used it to duplicate writing styles. Having it kick off the beginning without a planned out ending and just having it ramble on is just asking for crap, whether it be from an LLM or a graduate student writing a paper. I'm not saying it should be obvious because I only learned this from trial and error. It just won't compose an article with a deliberate structure if you ask it to just start writing it. It'll ramble and pack it with filler, in my experience. It'll also make frequent use of ineloquent and unnecessary transition clauses, such as "Finally,", or "Subsequently,", or "On the other hand," unless each paragraph is made independently and zooms out while following an outline. There might be another way, but telling it to just write the first chapter of a new James Joyce novel is just asking for rambling crap. On the other hand, asking it rewrite the first 5 pages of another novel in James Joyce's style with example writing given in-context will yield something much more coherent.
- kromem 3y ago
  This is also related to human composition. Whether written out or not, few writers just start typing a story and see where it goes. There was actually a great neuroscience paper last year looking at how human predictive processing seemed best categorized as a three layer approach - broad, medium, narrow temporality. And discussing the idea that this might apply to LLMs . So if you have a paragraph summary of a story you want to be 10 pages, having the first step be generating a one page treatment of the story from the paragraph and then feeding that in to writing the full story will lead to a better pacing. Want more interesting character arcs? Have it generate short character motivations, vulnerabilities, etc for key characters that gets fed in before the one page step. "Fancy autocomplete" is a powerful tool, but it isn't a replacement for story workshoping. But the latter can be embedded as a process into the former fairly successfully, particularly with few shot intermediate steps.
kromem 3y ago
Let's say you want an LLM to generate product descriptions. You might have a first pass where you generate a handful (each run will be slightly different). Then you can throw those against a pass where you give a system message of an in market shopper matching your target demo asking for the points that are most relevant to making a decision. Then you give the descriptions and bullet points to a final pass with a system message as an expert copywriter summarizing the multiple descriptions with an emphasis on the bullet points. You'll have MUCH better results than anything you'd be able to generate in a single pass. You'll need to play with zero shot vs few shot in each step, and there's other little hacks for consistency here and there that would be relevant, but a multipass approach like this makes a world of difference. As for code, it'd certainly be worth a try of taking the output from pass 1 and then running it though another pass "Can you refactor this code with an emphasis on reducing cyclomatic completely while improving readability and its future maintenance?" Though in general I'd love to see a dedicated refactoring fine tuned layer (or language specific layers) as an offering in the next year or so, as it's a task nuanced enough I don't know that a general purpose model is going to be great. But yes, I could definitely see running initial code generations though a second pass for improving the results.

jakderrida 3y ago

> Is it a good idea to do multiple passes when asking for code? Probably. However, I see their point when it comes to articles; having used it to duplicate writing styles. Having it kick off the beginning without a planned out ending and just having it ramble on is just asking for crap, whether it be from an LLM or a graduate student writing a paper. I'm not saying it should be obvious because I only learned this from trial and error. It just won't compose an article with a deliberate structure if you ask it to just start writing it. It'll ramble and pack it with filler, in my experience. It'll also make frequent use of ineloquent and unnecessary transition clauses, such as "Finally,", or "Subsequently,", or "On the other hand," unless each paragraph is made independently and zooms out while following an outline. There might be another way, but telling it to just write the first chapter of a new James Joyce novel is just asking for rambling crap. On the other hand, asking it rewrite the first 5 pages of another novel in James Joyce's style with example writing given in-context will yield something much more coherent.

kromem 3y ago
This is also related to human composition. Whether written out or not, few writers just start typing a story and see where it goes. There was actually a great neuroscience paper last year looking at how human predictive processing seemed best categorized as a three layer approach - broad, medium, narrow temporality. And discussing the idea that this might apply to LLMs . So if you have a paragraph summary of a story you want to be 10 pages, having the first step be generating a one page treatment of the story from the paragraph and then feeding that in to writing the full story will lead to a better pacing. Want more interesting character arcs? Have it generate short character motivations, vulnerabilities, etc for key characters that gets fed in before the one page step. "Fancy autocomplete" is a powerful tool, but it isn't a replacement for story workshoping. But the latter can be embedded as a process into the former fairly successfully, particularly with few shot intermediate steps.

kromem 3y ago

Let's say you want an LLM to generate product descriptions. You might have a first pass where you generate a handful (each run will be slightly different). Then you can throw those against a pass where you give a system message of an in market shopper matching your target demo asking for the points that are most relevant to making a decision. Then you give the descriptions and bullet points to a final pass with a system message as an expert copywriter summarizing the multiple descriptions with an emphasis on the bullet points. You'll have MUCH better results than anything you'd be able to generate in a single pass. You'll need to play with zero shot vs few shot in each step, and there's other little hacks for consistency here and there that would be relevant, but a multipass approach like this makes a world of difference. As for code, it'd certainly be worth a try of taking the output from pass 1 and then running it though another pass "Can you refactor this code with an emphasis on reducing cyclomatic completely while improving readability and its future maintenance?" Though in general I'd love to see a dedicated refactoring fine tuned layer (or language specific layers) as an offering in the next year or so, as it's a task nuanced enough I don't know that a general purpose model is going to be great. But yes, I could definitely see running initial code generations though a second pass for improving the results.