6 ms·

GPT-3 Creative Fiction (2020)

83 points by ijidak 4y ago

pmoriarty 4y ago
I've found Anthropic's Claude[1] to be far better for creative writing than GPT4 and even Claude+ [1] - https://poe.com/Claude-instant https://poe.com/Claude-instant
- hello_hny 4y ago
  Since when do you feel like this? I tested Turbo-Claude a month ago and wasn't impressed
  - pmoriarty 4y ago
    I don't know what "Turbo-Claude" is, but, like I said, in my experience Claude+ is worse than plain Claude (aka "Claude-Instant") at writing creative works.
- antifa 4y ago
  I'd like to be presented with more information before a random website demands my phone number.
  - pmoriarty 4y ago
    You don't have to give it your phone number. All you need is an email.
- gwern 4y ago
  I really need to try Claude, but, well, so much to keep up with these days... FWIW, I've found that GPT-4 in the Playground is pretty good at creative text compared to ChatGPT-3, with a bit of prompting. It's still sometimes inflexible, but mostly works. (GPT-4 inherits a lot of the weaknesses of GPT-3 for creative text, unfortunately; you can fool yourself into thinking that it's genuinely learned to rhyme or what phonetics are, but underneath, it's still the same BPE-flaws.) I am still working on some projects there - I'm most excited about an experiment into translating John Milton's _Paradise Lost_ into contemporary alliterative English verse, using an inner-monologue approach to revise it line by line repeatedly: https://www.reddit.com/r/MachineLearning/comments/12pqqg6/discussion_translation_of_japanese_to_english/jgnvay6/ https://www.reddit.com/r/MachineLearning/comments/12pqqg6/di... Sample of intro 'translation': https://pastebin.com/WmHKD9RP https://pastebin.com/WmHKD9RP GPT-4 suggested titling it _Perished Paradise_, and I rather like it. :)
orost 4y ago
I really want a high powered instruction-following LLM that's capable of assisting with writing fiction. GPT-3.5-turbo is fantastically capable at this if you can get it to drop its stilted AI-assistant voice, but doing so is too hard and too unreliable to be practical. I presume GPT-4 is even better but even harder to convince to try. It's frustrating to see through those glimpses that this capability exists, but has been trampled over and made mostly inaccessible in pursuit of making sure the tool doesn't ever offend by producing something that's not baby food. And all the open replication efforts are aping this approach. Sigh.
- james-revisoai 4y ago
  I agree. It's like instruct = Gain cosistentcy, lose creativity. Base = No consistency (really), but creative. Could a model be trained or positioned to be in the middle of these two?
  - orost 4y ago
    I suspect you could train a model to just shut up and follow instructions. I.e. instead of "Do X -> Sure! As a large language model, I'd love to help you with X!", just "Do X -> X". This would avoid giving the model a dull default voice. But it wouldn't be sufficiently hedged and "safe".
simse 4y ago
25 fonts loaded! That might be a record. But seriously, very interesting stuff.
- gwern 4y ago
  Yes, we were surprised too. 25! Wait, '25'? We don't have 25 fonts... Part of that is due to a bug we hadn't noticed in the CSS versioning code & have fixed. The actual number is more like 16 fonts (~500kb total), excluding the decorative dropcaps which are only 4kb each. This is one of the largest & most complex pages on gwern.net (even after splitting out much of it to gwern.net/gpt-3-nonfiction ), so, it pulls down pretty much every font. (Even the backup fonts, because of the emoji.) As for why we use those fonts, well, it's not like we didn't consider using just system fonts, but you run into problems as soon as you have any ambitions: https://gwern.net/design#webfonts https://gwern.net/design#webfonts
- cypherpunks01 4y ago
  Kanzlei-Initialen-T.ttf is beautiful!
  - gwern 4y ago
    If you like that, check out 'yinit' too: https://www.tug.org/TUGboat/tb12-1/tb31hara.pdf#page=8 https://www.tug.org/TUGboat/tb12-1/tb31hara.pdf#page=8
- examplary_cable 4y ago
  Would you rather have it be a plain text website? No attack, just wondering.
  - nickthegreek 4y ago
    I don’t think 25 fonts has ever made something better. I’m sure there is a number greater than 1 but less than 25 where we can read happily.
    - 4y ago
atleastoptimal 4y ago
He should do a follow-up on GPT-4. BTW, on a different note, would people consider Gwern the best writer in the rationalist community or are Yud and Slate Star Codex considered better?
- capableweb 4y ago
  What do you mean with "best writer"? One could argue that no writer is better than another, there is just people with different tastes.
  - atleastoptimal 4y ago
    There are objective measurements that people can agree on, even in subjective realms. In terms of popular impact, SSC obviously wins. However among "insiders" there are always implicitly lauded favorites. Is it about the general capacity to convey complex topics succinctly, general "immersion" in the writing and ability to consistently foster a compelling mind-space via the imagery and ideas handled in the text, etc? This is just me being a curious outsider. Interestingly I often peer into insular communities and find that there's often an unsaid tapestry of implicit assumptions and customs. All that I've discovered so far is that SSC is the darling favorite, Yud is liked but also made fun of for his behavior and more blunt approach, and Gwern is liked but not as notable, though imo he has the best website. Is the best way to peer into these ideas via a prolonged investigative type outside observation, or should one feel only a worthy potential to fit in to the common character of the crowd if they can take a short peek at the general shape of the culture and immediately "get it"?
    - pram 4y ago
      Maybe original, independent, and interesting thoughts have value on their own beyond if they potentially get you a chair in the cool kids lunch table.
  - examplary_cable 4y ago
    Sure, but I think when compared 1-to-1 some people are generally perceived to be better writs ... I'm not sure if GPT-4 would be significantly better than GPT-3 ... might depend on prompt and other issues.
Der_Einzige 4y ago
My work on constrained text generation / filter assisted decoding for LLMs is cited in this article! One of my proudest moments was being noticed by my senpai Gwern! https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai https://paperswithcode.com/paper/most-language-models-can-be... I want to update that just because GPT-4 appears to be far better at following constraints, doesn't mean that it's anywhere near perfect at following them. It's better now at my easy example of "ban the letter e" but if you ask for several constraints, or mixing lexical and phonetic constraints, it gets pretty awful pretty quickly. Filter assisted decoding can make any LLM (no matter how awful they are) follow constraints perfectly. I can't wait to get someone whose better at coding than me to implement these techniques in the major LLM frontends (oogabooga, llamma.ccp, etc) since my attempt at it was quite poopy research code: https://github.com/hellisotherpeople/constrained-text-generation-studio https://github.com/hellisotherpeople/constrained-text-genera... - especially with the idea of this but with all LLM types (e.g. bert), using a declarative syntax like langchain, and also supporting constrained beam search for sequence level constraints. BTW - just to motivate the nay-sayers on why filtering LLM vocabularies may be a cool (and dangerous) technique even in a world where LLMs became "perfect" at following lexical, semantic, or phonetic constraints in the prompt. Imagine asking ChatGPT to "generate some social security numbers". Now imagine doing that again when it's only allowed to generate number or hyphens. This is one of the many reasons why full vocabulary probability distributions are not exposed by folks like OpenAI
- thazework 4y ago
  Constrained generation i've found to produce the most creative-like results. Example: The stench of smoke and blood filled their nostrils as they crouched behind a fallen tree, its splintered trunk providing the only cover from the relentless hail of bullets, their heart hammering in their chest, threatening to burst free, as sweat mingled with dirt on their face, catching the frightened gaze of their comrades, their eyes wide with terror yet burning with determination, knowing they were all in this together with no turning back, seizing the opportunity as the gunfire briefly subsided, yelling "Move up! We've got to take that hill!" their voice breaking through the deafening cacophony of war, breaking cover without hesitation, boots pounding the earth as they ran through the hellish landscape, explosions erupting all around, the concussive force rattling their bones and sending tremors through the ground beneath their feet, seeing a fellow soldier fall, his anguished cry piercing their soul, diving to his side, dragging him into the meager cover of a shallow trench, his blood staining their hands, no time to mourn, continuing their charge, every muscle screaming in protest, lungs pleading for air, cresting the hill, the enemy bunker coming into view, sprinting towards it with a final surge of adrenaline, their hand shaking as they lobbed a grenade inside, the explosion tearing through the bunker, debris flying like deadly shrapnel, winning this small battle, but the weight of the war still pressing down on their shoulders, gazing out at the smoke-choked battlefield, wondering how many more battles they'd have to endure before the war was truly over.
dang 4y ago
Related: GPT-3 Creative Fiction - https://news.ycombinator.com/item?id=23722635 https://news.ycombinator.com/item?id=23722635 - July 2020 (97 comments) GPT-3 - https://news.ycombinator.com/item?id=23623845 https://news.ycombinator.com/item?id=23623845 - June 2020 (200 comments)

83 points by ijidak 4y ago

I've found Anthropic's Claude[1] to be far better for creative writing than GPT4 and even Claude+ [1] - https://poe.com/Claude-instant https://poe.com/Claude-instant

hello_hny 4y ago
Since when do you feel like this? I tested Turbo-Claude a month ago and wasn't impressed
- pmoriarty 4y ago
  I don't know what "Turbo-Claude" is, but, like I said, in my experience Claude+ is worse than plain Claude (aka "Claude-Instant") at writing creative works.
antifa 4y ago
I'd like to be presented with more information before a random website demands my phone number.
- pmoriarty 4y ago
  You don't have to give it your phone number. All you need is an email.
gwern 4y ago
I really need to try Claude, but, well, so much to keep up with these days... FWIW, I've found that GPT-4 in the Playground is pretty good at creative text compared to ChatGPT-3, with a bit of prompting. It's still sometimes inflexible, but mostly works. (GPT-4 inherits a lot of the weaknesses of GPT-3 for creative text, unfortunately; you can fool yourself into thinking that it's genuinely learned to rhyme or what phonetics are, but underneath, it's still the same BPE-flaws.) I am still working on some projects there - I'm most excited about an experiment into translating John Milton's _Paradise Lost_ into contemporary alliterative English verse, using an inner-monologue approach to revise it line by line repeatedly: https://www.reddit.com/r/MachineLearning/comments/12pqqg6/discussion_translation_of_japanese_to_english/jgnvay6/ https://www.reddit.com/r/MachineLearning/comments/12pqqg6/di... Sample of intro 'translation': https://pastebin.com/WmHKD9RP https://pastebin.com/WmHKD9RP GPT-4 suggested titling it _Perished Paradise_, and I rather like it. :)

orost 4y ago

I really want a high powered instruction-following LLM that's capable of assisting with writing fiction. GPT-3.5-turbo is fantastically capable at this if you can get it to drop its stilted AI-assistant voice, but doing so is too hard and too unreliable to be practical. I presume GPT-4 is even better but even harder to convince to try. It's frustrating to see through those glimpses that this capability exists, but has been trampled over and made mostly inaccessible in pursuit of making sure the tool doesn't ever offend by producing something that's not baby food. And all the open replication efforts are aping this approach. Sigh.

james-revisoai 4y ago
I agree. It's like instruct = Gain cosistentcy, lose creativity. Base = No consistency (really), but creative. Could a model be trained or positioned to be in the middle of these two?
- orost 4y ago
  I suspect you could train a model to just shut up and follow instructions. I.e. instead of "Do X -> Sure! As a large language model, I'd love to help you with X!", just "Do X -> X". This would avoid giving the model a dull default voice. But it wouldn't be sufficiently hedged and "safe".

simse 4y ago

25 fonts loaded! That might be a record. But seriously, very interesting stuff.

gwern 4y ago
Yes, we were surprised too. 25! Wait, '25'? We don't have 25 fonts... Part of that is due to a bug we hadn't noticed in the CSS versioning code & have fixed. The actual number is more like 16 fonts (~500kb total), excluding the decorative dropcaps which are only 4kb each. This is one of the largest & most complex pages on gwern.net (even after splitting out much of it to gwern.net/gpt-3-nonfiction ), so, it pulls down pretty much every font. (Even the backup fonts, because of the emoji.) As for why we use those fonts, well, it's not like we didn't consider using just system fonts, but you run into problems as soon as you have any ambitions: https://gwern.net/design#webfonts https://gwern.net/design#webfonts
cypherpunks01 4y ago
Kanzlei-Initialen-T.ttf is beautiful!
- gwern 4y ago
  If you like that, check out 'yinit' too: https://www.tug.org/TUGboat/tb12-1/tb31hara.pdf#page=8 https://www.tug.org/TUGboat/tb12-1/tb31hara.pdf#page=8
examplary_cable 4y ago
Would you rather have it be a plain text website? No attack, just wondering.
- nickthegreek 4y ago
  I don’t think 25 fonts has ever made something better. I’m sure there is a number greater than 1 but less than 25 where we can read happily.
  - 4y ago

atleastoptimal 4y ago

He should do a follow-up on GPT-4. BTW, on a different note, would people consider Gwern the best writer in the rationalist community or are Yud and Slate Star Codex considered better?

capableweb 4y ago
What do you mean with "best writer"? One could argue that no writer is better than another, there is just people with different tastes.
- atleastoptimal 4y ago
  There are objective measurements that people can agree on, even in subjective realms. In terms of popular impact, SSC obviously wins. However among "insiders" there are always implicitly lauded favorites. Is it about the general capacity to convey complex topics succinctly, general "immersion" in the writing and ability to consistently foster a compelling mind-space via the imagery and ideas handled in the text, etc? This is just me being a curious outsider. Interestingly I often peer into insular communities and find that there's often an unsaid tapestry of implicit assumptions and customs. All that I've discovered so far is that SSC is the darling favorite, Yud is liked but also made fun of for his behavior and more blunt approach, and Gwern is liked but not as notable, though imo he has the best website. Is the best way to peer into these ideas via a prolonged investigative type outside observation, or should one feel only a worthy potential to fit in to the common character of the crowd if they can take a short peek at the general shape of the culture and immediately "get it"?
  - pram 4y ago
    Maybe original, independent, and interesting thoughts have value on their own beyond if they potentially get you a chair in the cool kids lunch table.
- examplary_cable 4y ago
  Sure, but I think when compared 1-to-1 some people are generally perceived to be better writs ... I'm not sure if GPT-4 would be significantly better than GPT-3 ... might depend on prompt and other issues.

Der_Einzige 4y ago

My work on constrained text generation / filter assisted decoding for LLMs is cited in this article! One of my proudest moments was being noticed by my senpai Gwern! https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai https://paperswithcode.com/paper/most-language-models-can-be... I want to update that just because GPT-4 appears to be far better at following constraints, doesn't mean that it's anywhere near perfect at following them. It's better now at my easy example of "ban the letter e" but if you ask for several constraints, or mixing lexical and phonetic constraints, it gets pretty awful pretty quickly. Filter assisted decoding can make any LLM (no matter how awful they are) follow constraints perfectly. I can't wait to get someone whose better at coding than me to implement these techniques in the major LLM frontends (oogabooga, llamma.ccp, etc) since my attempt at it was quite poopy research code: https://github.com/hellisotherpeople/constrained-text-generation-studio https://github.com/hellisotherpeople/constrained-text-genera... - especially with the idea of this but with all LLM types (e.g. bert), using a declarative syntax like langchain, and also supporting constrained beam search for sequence level constraints. BTW - just to motivate the nay-sayers on why filtering LLM vocabularies may be a cool (and dangerous) technique even in a world where LLMs became "perfect" at following lexical, semantic, or phonetic constraints in the prompt. Imagine asking ChatGPT to "generate some social security numbers". Now imagine doing that again when it's only allowed to generate number or hyphens. This is one of the many reasons why full vocabulary probability distributions are not exposed by folks like OpenAI

thazework 4y ago
Constrained generation i've found to produce the most creative-like results. Example: The stench of smoke and blood filled their nostrils as they crouched behind a fallen tree, its splintered trunk providing the only cover from the relentless hail of bullets, their heart hammering in their chest, threatening to burst free, as sweat mingled with dirt on their face, catching the frightened gaze of their comrades, their eyes wide with terror yet burning with determination, knowing they were all in this together with no turning back, seizing the opportunity as the gunfire briefly subsided, yelling "Move up! We've got to take that hill!" their voice breaking through the deafening cacophony of war, breaking cover without hesitation, boots pounding the earth as they ran through the hellish landscape, explosions erupting all around, the concussive force rattling their bones and sending tremors through the ground beneath their feet, seeing a fellow soldier fall, his anguished cry piercing their soul, diving to his side, dragging him into the meager cover of a shallow trench, his blood staining their hands, no time to mourn, continuing their charge, every muscle screaming in protest, lungs pleading for air, cresting the hill, the enemy bunker coming into view, sprinting towards it with a final surge of adrenaline, their hand shaking as they lobbed a grenade inside, the explosion tearing through the bunker, debris flying like deadly shrapnel, winning this small battle, but the weight of the war still pressing down on their shoulders, gazing out at the smoke-choked battlefield, wondering how many more battles they'd have to endure before the war was truly over.

dang 4y ago

Related: GPT-3 Creative Fiction - https://news.ycombinator.com/item?id=23722635 https://news.ycombinator.com/item?id=23722635 - July 2020 (97 comments) GPT-3 - https://news.ycombinator.com/item?id=23623845 https://news.ycombinator.com/item?id=23623845 - June 2020 (200 comments)