3 ms·

What the difference between this and using chat history to concatenate outputs and prompting with something like “Now write the next section” repeatedly? I’ve d

What the difference between this and using chat history to concatenate outputs and prompting with something like “Now write the next section” repeatedly? I’ve done that with NotebookLM and it’ll write a complete fictional story based on sources, for example.

LeoPanthera 2y ago
Most LLMs are trained to write "complete" outputs. So each section will end up being like a tiny self-contained short book. Without manual editing they will not create long narratives.
- b33j0r 2y ago
  The notion that an LLM will be used or can be used mostly for a one-shot request/response has been one of the most idiosyncratic things about the first and second waves of this tech. Like, not only can it not “make a complete app in 45 seconds,” I almost never even want that.
dotnet00 2y ago
In my testing, that often causes the model to 'drift' and ramble wildly compared to just getting one long output from the very start. The issue is probably that when you split it by just asking for the next section, you're asking it to figure out how to continue from a block that wasn't written with the awareness that it'd have to add on to it. From the diagram on the repo, I guess this first plans out the structure for each block, and generates the blocks based on the plan.
thomasahle 2y ago
It would be the same if the model was "raw", trained only on text completion. But all models these days are RLHF'ed on (prompt, answer) pairs, so unfortunately they can get confused if the prompt already contains part of an answer.
- elfelf12 2y ago
  I think base models are far superior to those boring instruct tuned models. I would rather have a good text completionist than a chat bot. But as far as i know i am in a minority there.
ben_w 2y ago
My experience is that by default it's like that game where you fold some paper, draw just up to the fold, pass the paper to the next player and they continue, and then you unfold it and see what came out: https://i.kym-cdn.com/photos/images/facebook/001/782/166/960.jpg https://i.kym-cdn.com/photos/images/facebook/001/782/166/960... With editing or lots of regeneration it can still work, of course. (That said, I last tried this with the original ChatGPT…)

LeoPanthera 2y ago

Most LLMs are trained to write "complete" outputs. So each section will end up being like a tiny self-contained short book. Without manual editing they will not create long narratives.

b33j0r 2y ago
The notion that an LLM will be used or can be used mostly for a one-shot request/response has been one of the most idiosyncratic things about the first and second waves of this tech. Like, not only can it not “make a complete app in 45 seconds,” I almost never even want that.

dotnet00 2y ago

In my testing, that often causes the model to 'drift' and ramble wildly compared to just getting one long output from the very start. The issue is probably that when you split it by just asking for the next section, you're asking it to figure out how to continue from a block that wasn't written with the awareness that it'd have to add on to it. From the diagram on the repo, I guess this first plans out the structure for each block, and generates the blocks based on the plan.

thomasahle 2y ago

It would be the same if the model was "raw", trained only on text completion. But all models these days are RLHF'ed on (prompt, answer) pairs, so unfortunately they can get confused if the prompt already contains part of an answer.

elfelf12 2y ago
I think base models are far superior to those boring instruct tuned models. I would rather have a good text completionist than a chat bot. But as far as i know i am in a minority there.

ben_w 2y ago

My experience is that by default it's like that game where you fold some paper, draw just up to the fold, pass the paper to the next player and they continue, and then you unfold it and see what came out: https://i.kym-cdn.com/photos/images/facebook/001/782/166/960.jpg https://i.kym-cdn.com/photos/images/facebook/001/782/166/960... With editing or lots of regeneration it can still work, of course. (That said, I last tried this with the original ChatGPT…)