7 ms·

The prompt used: "Write a New Yorker-style story given the plot below. Make sure it is at least {{word_count}} words. Directly start with the story, do not say

by kromem 3y ago

The prompt used: "Write a New Yorker-style story given the plot below. Make sure it is at least {{word_count}} words. Directly start with the story, do not say things like ‘Here’s the story [...]" They didn't even bother using multiple passes to prompt better quality creative writing like "Write an outline for the story..." "Expand this outline focusing on improving style and tone..." "Here's a story written by an amateur writer. As a professional writer, give it a rewrite improving vocabulary, metaphor, and overall quality." As with a lot of the LLM stuff I see these days, I have to wonder at how much of what's being measured is the capacity of the tool and how much the measurement of the capacity of its users. I'd never imagine using a LLM zero shot in a single pass for any creative writing tasks, and measuring the inability to perform in suboptimal conditions isn't all that revealing or novel (pun intended).

joe_the_user 3y ago
If you are testing a system, you can't simultaneously tune the system - or at least, if you're establishing a new testing approach you can't once do. Maybe the testing approach is established, someone could experiment with ways to improve the system on the benchmark. But if you're establishing a benchmark, improving at the same time is going to make such a benchmark kind of meaningless.
- kromem 3y ago
  You tune it first. A better faith version of this should have been the experimenters creating a creative writing pipeline they found created the best output and then testing that. Using naive zero shot prompts in a single pass with no CoT doesn't even reflect real world production usage outside of the casual user of ChatGPT who discovered it a week ago. So yes, surprise surprise - casual and naive use of a tool isn't comparable to experts without the tool. And I'm sure my using a hammer to build a deck would perform much worse than an expert carpenter without a hammer. But that's saying less about the utility of the hammer than it is my ability to use the hammer.
hanselot 3y ago
Or my favourite thing to add as custom instruction: Rewrite this amateur's code like the lovechild of Molenaar (RIP) and van Rossum.
- kromem 3y ago
  One of the coolest prompt techniques I saw in research was asking "what expert would best be equipped to answer this task" followed with plugging that answer into "you are XYZ expert, answer this task."
bloopernova 3y ago
Is it a good idea to do multiple passes when asking for code? What sort of progression do we see in a multipass (!) workflow? Does anyone have any examples? (Wrangling AIs is a skill, I think, and thus learning more is always a good idea)
- jakderrida 3y ago
  > Is it a good idea to do multiple passes when asking for code? Probably. However, I see their point when it comes to articles; having used it to duplicate writing styles. Having it kick off the beginning without a planned out ending and just having it ramble on is just asking for crap, whether it be from an LLM or a graduate student writing a paper. I'm not saying it should be obvious because I only learned this from trial and error. It just won't compose an article with a deliberate structure if you ask it to just start writing it. It'll ramble and pack it with filler, in my experience. It'll also make frequent use of ineloquent and unnecessary transition clauses, such as "Finally,", or "Subsequently,", or "On the other hand," unless each paragraph is made independently and zooms out while following an outline. There might be another way, but telling it to just write the first chapter of a new James Joyce novel is just asking for rambling crap. On the other hand, asking it rewrite the first 5 pages of another novel in James Joyce's style with example writing given in-context will yield something much more coherent.
  - kromem 3y ago
    This is also related to human composition. Whether written out or not, few writers just start typing a story and see where it goes. There was actually a great neuroscience paper last year looking at how human predictive processing seemed best categorized as a three layer approach - broad, medium, narrow temporality. And discussing the idea that this might apply to LLMs . So if you have a paragraph summary of a story you want to be 10 pages, having the first step be generating a one page treatment of the story from the paragraph and then feeding that in to writing the full story will lead to a better pacing. Want more interesting character arcs? Have it generate short character motivations, vulnerabilities, etc for key characters that gets fed in before the one page step. "Fancy autocomplete" is a powerful tool, but it isn't a replacement for story workshoping. But the latter can be embedded as a process into the former fairly successfully, particularly with few shot intermediate steps.

by kromem 3y ago

joe_the_user 3y ago

If you are testing a system, you can't simultaneously tune the system - or at least, if you're establishing a new testing approach you can't once do. Maybe the testing approach is established, someone could experiment with ways to improve the system on the benchmark. But if you're establishing a benchmark, improving at the same time is going to make such a benchmark kind of meaningless.

kromem 3y ago
You tune it first. A better faith version of this should have been the experimenters creating a creative writing pipeline they found created the best output and then testing that. Using naive zero shot prompts in a single pass with no CoT doesn't even reflect real world production usage outside of the casual user of ChatGPT who discovered it a week ago. So yes, surprise surprise - casual and naive use of a tool isn't comparable to experts without the tool. And I'm sure my using a hammer to build a deck would perform much worse than an expert carpenter without a hammer. But that's saying less about the utility of the hammer than it is my ability to use the hammer.

hanselot 3y ago

Or my favourite thing to add as custom instruction: Rewrite this amateur's code like the lovechild of Molenaar (RIP) and van Rossum.

kromem 3y ago
One of the coolest prompt techniques I saw in research was asking "what expert would best be equipped to answer this task" followed with plugging that answer into "you are XYZ expert, answer this task."

bloopernova 3y ago

Is it a good idea to do multiple passes when asking for code? What sort of progression do we see in a multipass (!) workflow? Does anyone have any examples? (Wrangling AIs is a skill, I think, and thus learning more is always a good idea)

jakderrida 3y ago
> Is it a good idea to do multiple passes when asking for code? Probably. However, I see their point when it comes to articles; having used it to duplicate writing styles. Having it kick off the beginning without a planned out ending and just having it ramble on is just asking for crap, whether it be from an LLM or a graduate student writing a paper. I'm not saying it should be obvious because I only learned this from trial and error. It just won't compose an article with a deliberate structure if you ask it to just start writing it. It'll ramble and pack it with filler, in my experience. It'll also make frequent use of ineloquent and unnecessary transition clauses, such as "Finally,", or "Subsequently,", or "On the other hand," unless each paragraph is made independently and zooms out while following an outline. There might be another way, but telling it to just write the first chapter of a new James Joyce novel is just asking for rambling crap. On the other hand, asking it rewrite the first 5 pages of another novel in James Joyce's style with example writing given in-context will yield something much more coherent.
- kromem 3y ago
  This is also related to human composition. Whether written out or not, few writers just start typing a story and see where it goes. There was actually a great neuroscience paper last year looking at how human predictive processing seemed best categorized as a three layer approach - broad, medium, narrow temporality. And discussing the idea that this might apply to LLMs . So if you have a paragraph summary of a story you want to be 10 pages, having the first step be generating a one page treatment of the story from the paragraph and then feeding that in to writing the full story will lead to a better pacing. Want more interesting character arcs? Have it generate short character motivations, vulnerabilities, etc for key characters that gets fed in before the one page step. "Fancy autocomplete" is a powerful tool, but it isn't a replacement for story workshoping. But the latter can be embedded as a process into the former fairly successfully, particularly with few shot intermediate steps.