10 ms·

One of the biggest problems with hands off LLM writing (for long horizon stuff like novels) is that you can't really give them any details of your story because

One of the biggest problems with hands off LLM writing (for long horizon stuff like novels) is that you can't really give them any details of your story because they get absolutely neurotic with it. Imagine for instance you give the LLM the profile of the love interest for your epic fantasy, it will almost always have the main character meeting them within 3 pages (usually page 1) which is of course absolutely nonsensical pacing. No attempt to tell it otherwise changes anything. This is the first model that after 19 pages generated so far resembles anything like normal pacing even with a TON of details. I've never felt the need to generate anywhere near this much. Extremely impressed. Edit: Sharing it - https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221OgLP7Wfj3zP3-_bhc6U0GhtfJRSuDqqH%22%5D,%22action%22:%22open%22,%22userId%22:%22115416429261485925946%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%... with pastebin - https://pastebin.com/aiWuYcrF https://pastebin.com/aiWuYcrF

fshr 2y ago
I think you would be better off having the LLM help you build up the plot with high level chapter descriptions and then have it dig into each chapter or arc. Or start by giving it the beats before you ask it for help with specifics. That'd be better at keeping it on rails.
- famouswaffles 2y ago
  I don't disagree. Like with almost anything else involving LLMs, getting hands on produces better results but because in this instance, i much prefer to be the reader than the author or editor, it's really important to me that a LLM is capable of pacing long form writing properly on its own.
  - deleted 2y ago
    [deleted]
  - saberience 2y ago
    Random question, if you don't care about being a creator yourself, why do you even want to read long form writing written by an LLM? There are literally 10000s of actual human written books out there all of them better than anything an LLM can write, why not read them?
    - bufferoverflow 2y ago
      It's very hard to find good books written by humans. GoodReads is okay, but you quickly run out of high-end recommendations. I read mostly sci-fi, and the books that everyone recommends rarely end up being 10/10. But then I see some random recommendation on Reddit or HN, and it ends up being amazing. Human-generated slop is real.
      - eru 2y ago
        You could ask your LLM for a recommendation.
    - famouswaffles 2y ago
      Authors tell stories they want to tell and Readers read stories they want to read. The two don't necessarily overlap or overlap strongly enough. If you're even a little bit specific (nowhere near as specific as the above prompt, even just something like the dynamic between protagonists) then you don't actually have 10,000s of actual human written books. Not even close. Maybe it exists and maybe you'll find it good enough but if it's only been read by a few hundred or thousand people ? Good luck getting it recommended. I've read a LOT of fiction. I love reading. And if it's good enough, the idea of reading something created by a machine does not bother me at all. So of course i will continue to see if the machine is finally good enough and i can be a bit more specific.
    - Spooky23 2y ago
      Usually porn and fan fiction.
    - eru 2y ago
      > There are literally 10000s of actual human written books out there all of them better than anything an LLM can write, why not read them? 10000s is still much smaller than the space of possibilities for even a short prompt. You might be right that good human novels are better than what LLMs can manage today. But that's rapidly changing. And if you really need that Harry Potter / Superman / Three Musketeers crossover fan fiction itch scratched, you might not care that some other existing novel is 'better' in some abstract sense.
- tluyben2 2y ago
  That was what I tried on the train [0] a few weeks ago. I used Groq to get something very fast to see if it would work at least somewhat. It gives you a PDF in the end. Plugging in a better model gave much better results (still not really readable if you actually try to; at a glance it's convincing though), however, it was so slow that testing what kind of impossible. Cannot really have things done in parallel either because it does need to know what it pushed out before, at least the summary of it. [0] https://github.com/tluyben/bad-writer https://github.com/tluyben/bad-writer
Workaccount2 2y ago
Can you share it on a text sharing site? It seems you hit your share quota
- famouswaffles 2y ago
  Sure - https://pastebin.com/aiWuYcrF https://pastebin.com/aiWuYcrF
nout 2y ago
Using the AI in multiple phases is the approach that can handle this. Similarly to "Deep Research" approach - you can tell it to first generate a storyline with multiple twists and turns. Then ask the model to take this storyline and generate prompts for individual chapters. Then ask it to generate the individual chapters based on the prompts, etc.
- bbor 2y ago
  Yup -- asking a chatbot to create a novel in one shot is very similar to asking a human to improvise a novel in one shot.
  - famouswaffles 2y ago
    It's not a problem of one-shotting it. It's that the details cause a collapse. Even if you tried breaking it down which i have, you'd run into the same problem unless you tried holding its hand for every single page and then - what's the point ? I want to read the story not co-author it.
    - koakuma-chan 2y ago
      Doesn't novel literally mean something new? Can we really expect an LLM to produce a novel?
      - FloorEgg 2y ago
        Yes
      - eru 2y ago
        The etymology is pretty much irrelevant. In eg German, the word for novel is 'Roman'. But German readers don't expect their novels to be anymore romantic, nor do English readers expect their novels to be more novel. LLMs have been producing new things all the time. The question was always about quality of output, never about being able to produce anything new.
    - 2y ago
stephen_cagle 2y ago
Opening with "like a struck flint carried on a wind that wasn’t blowing." <chuckles> I don't know why, but that is just such a literal thing to say that it seems almost random.
comboy 2y ago
I like how critique of LLMs evolved on this site over the last few years. We are currently at nonsensical pacing while writing novels.
- ruraljuror 2y ago
  We are, if this comment is the standard for all criticism on this site. Your comment seems harsh. Perhaps novel writing is too low-brow of a standard for LLM critique?
  - jorl17 2y ago
    I didn't quite read parent's comment like that. I think it's more about how we keep moving the goalposts or, less cynically, how the models keep getting better and better. I am amazed at the progress that we are _still_ making on an almost monthly basis. It is unbelievable. Mind-boggling, to be honest. I am certain that the issue of pacing will be solved soon enough. I'd give 99% probability of it being solved in 3 years and 50% probability in 1.
    - jiggawatts 2y ago
      In my consulting career I sometimes get to tune database servers for performance. I have a bag of tricks that yield about +10-20% performance each. I get arguments about this from customers, typically along the lines of "that doesn't seem worth it." Yeah, but 10% plus 20% plus 20%... next thing you know you're at +100% and your server is literally double the speed! AI progress feels the same. Each little incremental improvement alone doesn't blow my skirt up, but we've had years of nearly monthly advances that have added up to something quite substantial.
      - eru 2y ago
        Yes, if you are Mary Poppins, each individual trick in your bag doesn't have to be large. (For those too young or unfamiliar: Mary Poppins famously had a bag that she could keep pulling things out of.)
      - 2y ago
solardev 2y ago
19 pages?! Am I the only one who prefers an AI that jumps straight to the point? - Buildup and happy background world-building - Subtle foreshadowing - Orcs attack - Hero is saved by unlikely warrior of astounding beauty - Evil is defeated until sales justify unnecessary sequel That's the kind of story fit for the modern attention span...
sagarpatil 2y ago
My prompt is nowhere near yours. Just for fun: Asked it to rewrite the first page of ‘The Fountainhead’ where Howard is a computer engineer, the rewrite is hilarious lol. https://gist.github.com/sagarspatil/e0b5443132501a3596c3a9a28571e97a https://gist.github.com/sagarspatil/e0b5443132501a3596c3a9a2...
deng 2y ago
I have actually read it and agree it is impressive. I will not comment much on the style of the writing, since this is very much subjective, but I would rate it as the "typical" modern fantasy style, which aims at filling as much pages as possible: very "flowery" language, lots of adjectives/adverbs, lots of details, lots of high-school prose ("Panic was a luxury they couldn't afford"). Not a big fan of that since I really miss the time where authors could write single, self-contained books instead of a sprawling series over thousands of pages, but I know of course that this kind of thing is very successful and people seem to enjoy it. If someone would give me this I would advise them to get a good copy editor. There are some logical inconsistencies, though. For instance, when they both enter the cellar through a trapdoor, Kael goes first, but the innkeeper instructs him to close the trapdoor behind them, which makes no sense. Also, Kael goes down the stairs and "risks a quick look back up" and can somehow see the front door bulging and the chaos outside through the windows, which obviously is impossible when you look up through a trapdoor, not to mention that previously it was said this entry is behind the bar counter, surely blocking the sight. Kael lights an oily rag which somehow becomes a torch. There's more generic things, like somehow these Eldertides being these mythical things no one has ever seen, yet they seem to be pretty common occurrences? The dimensions of the cellar are completely unclear, at first it seems to be very small but yet they move around it quite a bit. There's other issues, like people using the same words as the narrator ("the ooze"), like they listen to him. The inkeeper suddenly calling Kael by his name like they already know each other. Anyway, I would rate it "first draft". Of course, it is unclear whether the LLM would manage to write a consistent book, but I can fully believe that it would manage. I probably wouldn't want to read it.
- hjnilsson 2y ago
  Thank you for taking the time to do a thorough read, I just skimmed it, and the prose is certainly not for me. To me it lacks focus, but as you say, this may be the style the readers enjoy. And it also, as you say, really reuses words. Just reading I notice "phosphorescence" 4 times for example in this chapter, or "ooze" 17 times (!). It is very impressive though that it can create a somewhat cohesive storyline, and certainly an improvement over previous models.
- famouswaffles 2y ago
smcleod 2y ago
I have never used an LLM for fictional writing, but I have been writing large amounts of code with them for years. What I'd recommend is when you're defining your plan up front as to the sections of the content, simply state in which phase / chapter of the content they should meet. Planning generated content is often more important to invest in than the writing of it. Looking at your paste, your prompt is short and basic, it should probably be broken up into clear, formatted sections (try directives inside XML style tags). For such a large output as you're expecting id expect a considerable prompt of rules and context setting (maybe a page or two).
harrison_clarke 2y ago
this seems like something that planning would fix. i wonder if that's how it's doing it like, if it decides to <think> a table of contents, or chapter summaries, rather than just diving in at page 1
bwb 2y ago
why would you ever want to write a novel with AI, that is human stuff right? :)
- qingcharles 2y ago
  I'm terrible at writing, but I love reading. I've got ideas for novels, but I struggle to put them down. What I have found that works is to give the LLM the "world" outline at the beginning and then just feed it one line summary of each chapter and get it to write a chapter at a time. The problem is that the quality of results drastically decreases as the context length increases. After about 10 chapters the dialogue will start to get real snippy. I've tried getting it to summarize all the previous chapters and feed that back in, but it never includes enough detail.
  - bwb 2y ago
    The only way to get better at something is to do it. Start writing short stories or small novels, and you will get there over time. You don't even have to be a great writer to write a great book as well :). It helps, but readers will forgive a lot along your journey. Brandon Sanderson has a great series of lectures on how he approaches it that are awesome -> https://www.youtube.com/playlist?list=PLSH_xM-KC3ZvzkfVo_Dls0B5GiE2oMcLY https://www.youtube.com/playlist?list=PLSH_xM-KC3ZvzkfVo_Dls... You will get so many mental benefits from writing, too. I promise it is worth it. AI is a great tool if you hit a block and need to brainstorm.
    - qingcharles 2y ago
      No, you are absolutely right. A lot of the things people think they can't do are literally just lack of practice. My other problem is... lack of time :)
      - bwb 2y ago
        ack, I also have this problem :) I am working on some world-building for something I want to write one day, but I am trying just to write little things to help. I write a lot of nonfiction stuff for work, but I am worried that it might not translate as well to characters...
didip 2y ago
Give it time, this will be solved. I envisioned that one day, a framework will be created that can persist LLM current state on disk and then "fragments of memories" can be paged in and out into memory. When that happened, LLM will be able to remember everything.
dyauspitr 2y ago
That is mind blowing. To this fantasy reader that’s pure magic.
pantsforbirds 2y ago
I had Grok summarize + evaluate the first chapter with thinking mode enabled. The output was actually pretty solid: https://pastebin.com/pLjHJF8E https://pastebin.com/pLjHJF8E. I wouldn't be surprised if someone figured out a solid mixture of models working as a writer (team of writers?) + editor(s) and managed to generate a full book from it. Maybe some mixture of general outlining + maintaining a wiki with a basic writing and editing flow would be enough. I think you could probably find a way to maintain plot consistency, but I'm not so sure about maintaining writing style.

fshr 2y ago

I think you would be better off having the LLM help you build up the plot with high level chapter descriptions and then have it dig into each chapter or arc. Or start by giving it the beats before you ask it for help with specifics. That'd be better at keeping it on rails.

famouswaffles 2y ago
I don't disagree. Like with almost anything else involving LLMs, getting hands on produces better results but because in this instance, i much prefer to be the reader than the author or editor, it's really important to me that a LLM is capable of pacing long form writing properly on its own.
- deleted 2y ago
  [deleted]
- saberience 2y ago
  Random question, if you don't care about being a creator yourself, why do you even want to read long form writing written by an LLM? There are literally 10000s of actual human written books out there all of them better than anything an LLM can write, why not read them?
  - bufferoverflow 2y ago
    It's very hard to find good books written by humans. GoodReads is okay, but you quickly run out of high-end recommendations. I read mostly sci-fi, and the books that everyone recommends rarely end up being 10/10. But then I see some random recommendation on Reddit or HN, and it ends up being amazing. Human-generated slop is real.
    - eru 2y ago
      You could ask your LLM for a recommendation.
  - famouswaffles 2y ago
    Authors tell stories they want to tell and Readers read stories they want to read. The two don't necessarily overlap or overlap strongly enough. If you're even a little bit specific (nowhere near as specific as the above prompt, even just something like the dynamic between protagonists) then you don't actually have 10,000s of actual human written books. Not even close. Maybe it exists and maybe you'll find it good enough but if it's only been read by a few hundred or thousand people ? Good luck getting it recommended. I've read a LOT of fiction. I love reading. And if it's good enough, the idea of reading something created by a machine does not bother me at all. So of course i will continue to see if the machine is finally good enough and i can be a bit more specific.
  - Spooky23 2y ago
    Usually porn and fan fiction.
  - eru 2y ago
    > There are literally 10000s of actual human written books out there all of them better than anything an LLM can write, why not read them? 10000s is still much smaller than the space of possibilities for even a short prompt. You might be right that good human novels are better than what LLMs can manage today. But that's rapidly changing. And if you really need that Harry Potter / Superman / Three Musketeers crossover fan fiction itch scratched, you might not care that some other existing novel is 'better' in some abstract sense.
tluyben2 2y ago
That was what I tried on the train [0] a few weeks ago. I used Groq to get something very fast to see if it would work at least somewhat. It gives you a PDF in the end. Plugging in a better model gave much better results (still not really readable if you actually try to; at a glance it's convincing though), however, it was so slow that testing what kind of impossible. Cannot really have things done in parallel either because it does need to know what it pushed out before, at least the summary of it. [0] https://github.com/tluyben/bad-writer https://github.com/tluyben/bad-writer

Workaccount2 2y ago

Can you share it on a text sharing site? It seems you hit your share quota

famouswaffles 2y ago
Sure - https://pastebin.com/aiWuYcrF https://pastebin.com/aiWuYcrF

nout 2y ago

Using the AI in multiple phases is the approach that can handle this. Similarly to "Deep Research" approach - you can tell it to first generate a storyline with multiple twists and turns. Then ask the model to take this storyline and generate prompts for individual chapters. Then ask it to generate the individual chapters based on the prompts, etc.

bbor 2y ago
Yup -- asking a chatbot to create a novel in one shot is very similar to asking a human to improvise a novel in one shot.
- famouswaffles 2y ago
  It's not a problem of one-shotting it. It's that the details cause a collapse. Even if you tried breaking it down which i have, you'd run into the same problem unless you tried holding its hand for every single page and then - what's the point ? I want to read the story not co-author it.
  - koakuma-chan 2y ago
    Doesn't novel literally mean something new? Can we really expect an LLM to produce a novel?
    - FloorEgg 2y ago
      Yes
    - eru 2y ago
      The etymology is pretty much irrelevant. In eg German, the word for novel is 'Roman'. But German readers don't expect their novels to be anymore romantic, nor do English readers expect their novels to be more novel. LLMs have been producing new things all the time. The question was always about quality of output, never about being able to produce anything new.
  - 2y ago

stephen_cagle 2y ago

Opening with "like a struck flint carried on a wind that wasn’t blowing." <chuckles> I don't know why, but that is just such a literal thing to say that it seems almost random.

comboy 2y ago

I like how critique of LLMs evolved on this site over the last few years. We are currently at nonsensical pacing while writing novels.

ruraljuror 2y ago
We are, if this comment is the standard for all criticism on this site. Your comment seems harsh. Perhaps novel writing is too low-brow of a standard for LLM critique?
- jorl17 2y ago
  I didn't quite read parent's comment like that. I think it's more about how we keep moving the goalposts or, less cynically, how the models keep getting better and better. I am amazed at the progress that we are _still_ making on an almost monthly basis. It is unbelievable. Mind-boggling, to be honest. I am certain that the issue of pacing will be solved soon enough. I'd give 99% probability of it being solved in 3 years and 50% probability in 1.
  - jiggawatts 2y ago
    In my consulting career I sometimes get to tune database servers for performance. I have a bag of tricks that yield about +10-20% performance each. I get arguments about this from customers, typically along the lines of "that doesn't seem worth it." Yeah, but 10% plus 20% plus 20%... next thing you know you're at +100% and your server is literally double the speed! AI progress feels the same. Each little incremental improvement alone doesn't blow my skirt up, but we've had years of nearly monthly advances that have added up to something quite substantial.
    - eru 2y ago
      Yes, if you are Mary Poppins, each individual trick in your bag doesn't have to be large. (For those too young or unfamiliar: Mary Poppins famously had a bag that she could keep pulling things out of.)
    - 2y ago

solardev 2y ago

19 pages?! Am I the only one who prefers an AI that jumps straight to the point? - Buildup and happy background world-building - Subtle foreshadowing - Orcs attack - Hero is saved by unlikely warrior of astounding beauty - Evil is defeated until sales justify unnecessary sequel That's the kind of story fit for the modern attention span...

sagarpatil 2y ago

My prompt is nowhere near yours. Just for fun: Asked it to rewrite the first page of ‘The Fountainhead’ where Howard is a computer engineer, the rewrite is hilarious lol. https://gist.github.com/sagarspatil/e0b5443132501a3596c3a9a28571e97a https://gist.github.com/sagarspatil/e0b5443132501a3596c3a9a2...

deng 2y ago

I have actually read it and agree it is impressive. I will not comment much on the style of the writing, since this is very much subjective, but I would rate it as the "typical" modern fantasy style, which aims at filling as much pages as possible: very "flowery" language, lots of adjectives/adverbs, lots of details, lots of high-school prose ("Panic was a luxury they couldn't afford"). Not a big fan of that since I really miss the time where authors could write single, self-contained books instead of a sprawling series over thousands of pages, but I know of course that this kind of thing is very successful and people seem to enjoy it. If someone would give me this I would advise them to get a good copy editor. There are some logical inconsistencies, though. For instance, when they both enter the cellar through a trapdoor, Kael goes first, but the innkeeper instructs him to close the trapdoor behind them, which makes no sense. Also, Kael goes down the stairs and "risks a quick look back up" and can somehow see the front door bulging and the chaos outside through the windows, which obviously is impossible when you look up through a trapdoor, not to mention that previously it was said this entry is behind the bar counter, surely blocking the sight. Kael lights an oily rag which somehow becomes a torch. There's more generic things, like somehow these Eldertides being these mythical things no one has ever seen, yet they seem to be pretty common occurrences? The dimensions of the cellar are completely unclear, at first it seems to be very small but yet they move around it quite a bit. There's other issues, like people using the same words as the narrator ("the ooze"), like they listen to him. The inkeeper suddenly calling Kael by his name like they already know each other. Anyway, I would rate it "first draft". Of course, it is unclear whether the LLM would manage to write a consistent book, but I can fully believe that it would manage. I probably wouldn't want to read it.

hjnilsson 2y ago
Thank you for taking the time to do a thorough read, I just skimmed it, and the prose is certainly not for me. To me it lacks focus, but as you say, this may be the style the readers enjoy. And it also, as you say, really reuses words. Just reading I notice "phosphorescence" 4 times for example in this chapter, or "ooze" 17 times (!). It is very impressive though that it can create a somewhat cohesive storyline, and certainly an improvement over previous models.
famouswaffles 2y ago

smcleod 2y ago

I have never used an LLM for fictional writing, but I have been writing large amounts of code with them for years. What I'd recommend is when you're defining your plan up front as to the sections of the content, simply state in which phase / chapter of the content they should meet. Planning generated content is often more important to invest in than the writing of it. Looking at your paste, your prompt is short and basic, it should probably be broken up into clear, formatted sections (try directives inside XML style tags). For such a large output as you're expecting id expect a considerable prompt of rules and context setting (maybe a page or two).

harrison_clarke 2y ago

this seems like something that planning would fix. i wonder if that's how it's doing it like, if it decides to <think> a table of contents, or chapter summaries, rather than just diving in at page 1

bwb 2y ago

why would you ever want to write a novel with AI, that is human stuff right? :)

qingcharles 2y ago
I'm terrible at writing, but I love reading. I've got ideas for novels, but I struggle to put them down. What I have found that works is to give the LLM the "world" outline at the beginning and then just feed it one line summary of each chapter and get it to write a chapter at a time. The problem is that the quality of results drastically decreases as the context length increases. After about 10 chapters the dialogue will start to get real snippy. I've tried getting it to summarize all the previous chapters and feed that back in, but it never includes enough detail.
- bwb 2y ago
  The only way to get better at something is to do it. Start writing short stories or small novels, and you will get there over time. You don't even have to be a great writer to write a great book as well :). It helps, but readers will forgive a lot along your journey. Brandon Sanderson has a great series of lectures on how he approaches it that are awesome -> https://www.youtube.com/playlist?list=PLSH_xM-KC3ZvzkfVo_Dls0B5GiE2oMcLY https://www.youtube.com/playlist?list=PLSH_xM-KC3ZvzkfVo_Dls... You will get so many mental benefits from writing, too. I promise it is worth it. AI is a great tool if you hit a block and need to brainstorm.
  - qingcharles 2y ago
    No, you are absolutely right. A lot of the things people think they can't do are literally just lack of practice. My other problem is... lack of time :)
    - bwb 2y ago
      ack, I also have this problem :) I am working on some world-building for something I want to write one day, but I am trying just to write little things to help. I write a lot of nonfiction stuff for work, but I am worried that it might not translate as well to characters...

didip 2y ago

Give it time, this will be solved. I envisioned that one day, a framework will be created that can persist LLM current state on disk and then "fragments of memories" can be paged in and out into memory. When that happened, LLM will be able to remember everything.

dyauspitr 2y ago

That is mind blowing. To this fantasy reader that’s pure magic.

pantsforbirds 2y ago

I had Grok summarize + evaluate the first chapter with thinking mode enabled. The output was actually pretty solid: https://pastebin.com/pLjHJF8E https://pastebin.com/pLjHJF8E. I wouldn't be surprised if someone figured out a solid mixture of models working as a writer (team of writers?) + editor(s) and managed to generate a full book from it. Maybe some mixture of general outlining + maintaining a wiki with a basic writing and editing flow would be enough. I think you could probably find a way to maintain plot consistency, but I'm not so sure about maintaining writing style.