5 ms·

Can you elaborate on how you are having GPT2 contribute to your comments? What is your process?

nullc 6y ago
I write a prompt, often copying text from the articles or other comments, and have it generate a lot of completions. I skim over the completions and grab interesting parts. For example, complaining about "quoted text" in my above comment was GPT2's suggestion (and also an actual issue with GPT2 which it was exhibiting by producing that text). Actually, all the text above from "There are a few" and beyond were written by GPT2. They were true enough, though more often I just extract the ideas and write my own text. In some contexts, but not on HN, I've also found it useful to get GPT2 to generate replies to my posts. It occasionally spots counter-arguments that I forgot to address and I can edit my comments to address them. Using computer text generation as a writing tool isn't new to me, I wrote my first computer poetry generator in the mid-90s. I've put out a few other things which were mostly machine generated. To be clear: GPT2 puts out a lot of junk-- mostly junk, in fact. But a lot beauty can be found sifting gems out of a sea of noise. Some authors have taken psychedelic drugs to enhance their creative processor, with GPT2 it is your word-processor that takes the drugs.
- modeless 6y ago
  > Some authors have taken psychedelic drugs to enhance their creative processor, with GPT2 it is your word-processor that takes the drugs. I love this quote. I hope it wasn't written by GPT-2.
- O_H_E 6y ago
  .....MAN, I am not sure how I feel about this. Don't take this the wrong way, it definitely looks like you are just using technology to produce art/craft in your own way. I will definitely look into this the next time I am writing cover letters — that I dread. Tools/sites/docker-images that improve your workflow? Damn, kudos again.
  - nullc 6y ago
    I just tried GPT2 at cover letters and I can't figure out a prompt that gets it to consistently write good ones, instead I can mostly get it to write bad ones (as in the sort of stuff you might see unsuccessful applicants submit). My example problem was to get it to apply to Y Combinator. It managed to misspell the organization which isn't something GPT2 normally does. Here are some examples: https://0bin.net/paste/RzN1n+cubZwvHDKc#hV5+TXf4eNRRB9ROuXIT7ILb22et7985WPc+A9zD6t1 https://0bin.net/paste/RzN1n+cubZwvHDKc#hV5+TXf4eNRRB9ROuXIT... After the first time it produced "Dear Paul Graham," I stuck it on the end of my prompt as it tended to reliably produce a cover letter (and one for YC) rather than a blank job application form. Other than throwing lots of computing power at it: I find that it's useful to include some largely irrelevant flavour text in your prefixes that you can stir until you hit on a combination that triggers the results you want (seemingly random things will send the model off in vastly different directions). When it has good initial output but goes off the rails to you can just terminate it at the last good part and continue from there-- a lot of GPT2's worst sins can be correct with fairly low effort that way. As mentioned, I think these examples are mostly pretty poor. I suspect that if I worked on it I could probably hit on a prompt that caused it to generate better cover letters... That said, there are perhaps a few things I might not have considered without reading them.
    - saberience 6y ago
      Maybe this is a dumb question, because you seem like someone who thinks in a radically different way to me. But why would you try and use GPT-2 to help you write better English versus actually learn to become a better writer yourself? I say this as someone who reads extensively and is considered a "good" writer and have won prizes before for my writing. I became a better writer through reading more books (especially classic literature) and writing more. Cribbing off a computer algorithm and copying and pasting seems like it would just be hurting your own growth, as well as being lazy to boot.
      - nullc 6y ago
        I don't care that much about being a 'good writer' at least not according to any abstract sense of good. I care more about exploring interesting ideas and interesting interactions, joyful leaps of insight, surprise and serendipity. Like in any other art just being different can have its own merit. The nearest I usually come to caring about good is that at times I aspire to be an /effective/ writer. But in many cases I've found being an effective writer can requires making the far-out leap that no one else was making or it can depend on making good guesses at the minds of readers far different from my own. While lucid language and fully formed ideas are necessary tools, the spice of something unfamiliarly or too familiar-- an allusion, even a silly or vulgar one, that people can't unstick from their minds, or anticipating their thoughts so that your writing speaks in harmony with their inner voice-- can really help a message stand out. Some of the most effective writing, as I see it, comes from taking a step back from the words on the page and asking yourself "what does this actually say about the world?" Any tool that helps you adopt a different perspective can be useful to these ends. Working with the machine is not just copy-and-paste-- if it were, there would already be no more need for writers. It is its own art and one that has yet to be mastered by anyone. In particular tools like GPT2 are absurdly sensitive to the prompts. There is also skill and creativity that can go into sampling, knowing when tell it to take greater or lesser risk, where to cut it off and retry or which word to insert to move it back in a fruitful direction. Fine details like what you name objects, people, and places implicitly signal to the model the genre of the writing and help guide it down useful paths or into blind allies. We sometimes think about the connotations of the words we use but for the model the connotations are all that matter. People talk some about bias in AI but in some sense the model is nothing but biases. Strange biases, some alien and impervious to analysis, others are biting social commentary once you notice them. Writers have always used devices and tools of various forms-- narrative forms, patterns of speech, constrained writing, and so on-- and machine text generation is but one more. I suppose that there are more dull ways of using it than brilliant ones, but that is true of any tool. We might write with a pen or a typewriter, but the tool doesn't become the author simply because we used it to write. The same is true using the computer even if, at times, the line may be fuzzier. We also shouldn't kid ourselves: Good writing, however we define it, is both derivative and also dependant on a health dose of chance and luck. If all the machine did was give us another way to be derivative or another way of finding fortunate statements by chance we could still find value in that alone. I don't think it's likely that I'll be the first to discover the best ways to use machine-co-authorship to improve writing, if such ways are ever to be discovered. Neither is it likely that I'll become a renowned writer by traditional means. But the former currently has less competition and the latter has already been done. And besides, I find it fun.
- richk449 6y ago
  Pretty much everyone agrees that these models, no matter how good they are at writing, don’t understand what they are writing, and can’t develop new ideas, only combine existing ideas together. Given that, it’s seems that what you are doing adds only noise, and no value, to HN. Of course, that is probably also true of a large fraction of pure-human posters ...
  - cryptoz 6y ago
    > Pretty much everyone agrees that these models, no matter how good they are at writing, don’t understand what they are writing, and can’t develop new ideas, only combine existing ideas together. Well I guess not everyone agrees with that, because I don't for example. First, how different are the two things, "develop new ideas" vs "combine existing ideas together"? Are we certain that some things that you consider new ideas aren't ever things I would consider the combining of existing ideas? Regardless, it seems to me that these language models may very well produce "new ideas" by chance, even if it doesn't itself 'recognize' that that is what it's doing. > Given that, it’s seems that what you are doing adds only noise, and no value, to HN. I don't agree that the only value added to HN is 'new ideas'. There are lots of old ideas that have lots of value in being communicated and discussed.
  - deleted 6y ago
    [deleted]
- rsync 6y ago
  "Can you elaborate on how you are having GPT2 contribute to your comments? What is your process?" ... "I write a prompt, often copying text from the articles or other comments, and have it generate a lot of completions." ... Can you elaborate, even further, with details about the actual UI of GPT2/3 (which I have never used, nor seen used) ? What I mean is ... when you "write a prompt", is that stdio on the command line ? Can you paste an example ? Do you, then, get a single "completion" as an stdio result and ... you can just get new ones by up-arrowing and repeating your command ? Am I getting warm here or am I stuck in a unix command centric paradigm of what this all looks like ?
  - nullc 6y ago
    More or less. I have tools I wrote that takes sampling settings and a string and dispatches it across a cluster of machines. Via cluster-ssh I see the each of them expanding the text in a different random sampling. This is custom stuff I've wrote that is tied to my environment. If you'd just like to play around, I can recommend https://bellard.org/nncp/gpt2tc.html https://bellard.org/nncp/gpt2tc.html in text generation mode as being extremely easy to get going. (the default model however is pretty small and dumb). The paradigm you're thinking about is exactly what you get from gpt2tc. At any point I can abort a job, tweak the sampling settings, or the text I'm expanding. E.g. one operation is that if I see one sample seems to be on a good path but has made an error, I'll abort it and restart all of them from a fixed version of that sample. Often I'll end up with my prompts in text files because they get a bit long at times, also escaping quotes and linebreaks on the commandline can be a pita. I have some aspirations of integrating this into a text editor, so that as I type future text is just appearing ahead of me and I can just hit a cursor to accept parts of it. But in my experience GPT2 isn't good enough where looking only at one continuation is enough or where I don't get a lot of advantage in having it work from modified text. GPT2 has preconceived notions about what kind of text you're writing based on the words you use. So it can be useful to alter your input text to replace persons/places/things names with different ones that get in into the right context and then back substitute them. To give a concrete example, if I wanted GPT2 to show me example bio blurbs for my partner (always a pain to write but easier if someone generates examples), it works better if I change her name--Kat-- because it either turns her into a man or it resists talking about her being a lawyer and a board member and instead makes her into an artist or a dancer. One thing to watch out for is that when GPT2 makes a benign error, like switching the gender of a pronoun mid-stream it often trashes the quality of the later output in unexpected ways (like causing it to output nonsense). Changing my SO's name to something it's not unsure about saves me time having to abort completions that have gone off the rails. Perhaps I shouldn't use a gendered example. GPT2 isn't sexist its everything-ist. Every word has 1001 hidden meanings that subtly bias its behaviour, many of the biases are actually the point- they exist in the world and they're what makes the output useful-- others are weird and unexpected and are just a training/corpus artifact. Good use requires a degree of anticipating and exploiting these biases. I even have to change my own name, because GPT2 knows very well that "Greg Maxwell" has something to do with Bitcoin and it readily lapses into Bitcoin conspiracy theories if my name is used. ["Gessh, even the machines are harassing me!"] Another class of biases is language that trips it into fantasy land or 'silly' writing. Any word that is commonly used in writing for children is at risk. If there is another common word that only adults use it might be a better choice if you want serious text, or you setup a context that makes the meaning more clear. It looks like for GPT3 you don't have to use subtle hints as much. E.g. that you can just tell it more explicitly what kind of thing you're doing and won't get tripped up as much by spurious correlations. OTOH, since it doesn't look like they're going to release the GPT3 model it's likely that I'll never get an opportunity to use it in my workflow.

I write a prompt, often copying text from the articles or other comments, and have it generate a lot of completions. I skim over the completions and grab interesting parts. For example, complaining about "quoted text" in my above comment was GPT2's suggestion (and also an actual issue with GPT2 which it was exhibiting by producing that text). Actually, all the text above from "There are a few" and beyond were written by GPT2. They were true enough, though more often I just extract the ideas and write my own text. In some contexts, but not on HN, I've also found it useful to get GPT2 to generate replies to my posts. It occasionally spots counter-arguments that I forgot to address and I can edit my comments to address them. Using computer text generation as a writing tool isn't new to me, I wrote my first computer poetry generator in the mid-90s. I've put out a few other things which were mostly machine generated. To be clear: GPT2 puts out a lot of junk-- mostly junk, in fact. But a lot beauty can be found sifting gems out of a sea of noise. Some authors have taken psychedelic drugs to enhance their creative processor, with GPT2 it is your word-processor that takes the drugs.

modeless 6y ago
> Some authors have taken psychedelic drugs to enhance their creative processor, with GPT2 it is your word-processor that takes the drugs. I love this quote. I hope it wasn't written by GPT-2.
O_H_E 6y ago
.....MAN, I am not sure how I feel about this. Don't take this the wrong way, it definitely looks like you are just using technology to produce art/craft in your own way. I will definitely look into this the next time I am writing cover letters — that I dread. Tools/sites/docker-images that improve your workflow? Damn, kudos again.
- nullc 6y ago
  I just tried GPT2 at cover letters and I can't figure out a prompt that gets it to consistently write good ones, instead I can mostly get it to write bad ones (as in the sort of stuff you might see unsuccessful applicants submit). My example problem was to get it to apply to Y Combinator. It managed to misspell the organization which isn't something GPT2 normally does. Here are some examples: https://0bin.net/paste/RzN1n+cubZwvHDKc#hV5+TXf4eNRRB9ROuXIT7ILb22et7985WPc+A9zD6t1 https://0bin.net/paste/RzN1n+cubZwvHDKc#hV5+TXf4eNRRB9ROuXIT... After the first time it produced "Dear Paul Graham," I stuck it on the end of my prompt as it tended to reliably produce a cover letter (and one for YC) rather than a blank job application form. Other than throwing lots of computing power at it: I find that it's useful to include some largely irrelevant flavour text in your prefixes that you can stir until you hit on a combination that triggers the results you want (seemingly random things will send the model off in vastly different directions). When it has good initial output but goes off the rails to you can just terminate it at the last good part and continue from there-- a lot of GPT2's worst sins can be correct with fairly low effort that way. As mentioned, I think these examples are mostly pretty poor. I suspect that if I worked on it I could probably hit on a prompt that caused it to generate better cover letters... That said, there are perhaps a few things I might not have considered without reading them.
  - saberience 6y ago
    Maybe this is a dumb question, because you seem like someone who thinks in a radically different way to me. But why would you try and use GPT-2 to help you write better English versus actually learn to become a better writer yourself? I say this as someone who reads extensively and is considered a "good" writer and have won prizes before for my writing. I became a better writer through reading more books (especially classic literature) and writing more. Cribbing off a computer algorithm and copying and pasting seems like it would just be hurting your own growth, as well as being lazy to boot.
    - nullc 6y ago
      I don't care that much about being a 'good writer' at least not according to any abstract sense of good. I care more about exploring interesting ideas and interesting interactions, joyful leaps of insight, surprise and serendipity. Like in any other art just being different can have its own merit. The nearest I usually come to caring about good is that at times I aspire to be an /effective/ writer. But in many cases I've found being an effective writer can requires making the far-out leap that no one else was making or it can depend on making good guesses at the minds of readers far different from my own. While lucid language and fully formed ideas are necessary tools, the spice of something unfamiliarly or too familiar-- an allusion, even a silly or vulgar one, that people can't unstick from their minds, or anticipating their thoughts so that your writing speaks in harmony with their inner voice-- can really help a message stand out. Some of the most effective writing, as I see it, comes from taking a step back from the words on the page and asking yourself "what does this actually say about the world?" Any tool that helps you adopt a different perspective can be useful to these ends. Working with the machine is not just copy-and-paste-- if it were, there would already be no more need for writers. It is its own art and one that has yet to be mastered by anyone. In particular tools like GPT2 are absurdly sensitive to the prompts. There is also skill and creativity that can go into sampling, knowing when tell it to take greater or lesser risk, where to cut it off and retry or which word to insert to move it back in a fruitful direction. Fine details like what you name objects, people, and places implicitly signal to the model the genre of the writing and help guide it down useful paths or into blind allies. We sometimes think about the connotations of the words we use but for the model the connotations are all that matter. People talk some about bias in AI but in some sense the model is nothing but biases. Strange biases, some alien and impervious to analysis, others are biting social commentary once you notice them. Writers have always used devices and tools of various forms-- narrative forms, patterns of speech, constrained writing, and so on-- and machine text generation is but one more. I suppose that there are more dull ways of using it than brilliant ones, but that is true of any tool. We might write with a pen or a typewriter, but the tool doesn't become the author simply because we used it to write. The same is true using the computer even if, at times, the line may be fuzzier. We also shouldn't kid ourselves: Good writing, however we define it, is both derivative and also dependant on a health dose of chance and luck. If all the machine did was give us another way to be derivative or another way of finding fortunate statements by chance we could still find value in that alone. I don't think it's likely that I'll be the first to discover the best ways to use machine-co-authorship to improve writing, if such ways are ever to be discovered. Neither is it likely that I'll become a renowned writer by traditional means. But the former currently has less competition and the latter has already been done. And besides, I find it fun.
richk449 6y ago
Pretty much everyone agrees that these models, no matter how good they are at writing, don’t understand what they are writing, and can’t develop new ideas, only combine existing ideas together. Given that, it’s seems that what you are doing adds only noise, and no value, to HN. Of course, that is probably also true of a large fraction of pure-human posters ...
- cryptoz 6y ago
  > Pretty much everyone agrees that these models, no matter how good they are at writing, don’t understand what they are writing, and can’t develop new ideas, only combine existing ideas together. Well I guess not everyone agrees with that, because I don't for example. First, how different are the two things, "develop new ideas" vs "combine existing ideas together"? Are we certain that some things that you consider new ideas aren't ever things I would consider the combining of existing ideas? Regardless, it seems to me that these language models may very well produce "new ideas" by chance, even if it doesn't itself 'recognize' that that is what it's doing. > Given that, it’s seems that what you are doing adds only noise, and no value, to HN. I don't agree that the only value added to HN is 'new ideas'. There are lots of old ideas that have lots of value in being communicated and discussed.
- deleted 6y ago
  [deleted]
rsync 6y ago
"Can you elaborate on how you are having GPT2 contribute to your comments? What is your process?" ... "I write a prompt, often copying text from the articles or other comments, and have it generate a lot of completions." ... Can you elaborate, even further, with details about the actual UI of GPT2/3 (which I have never used, nor seen used) ? What I mean is ... when you "write a prompt", is that stdio on the command line ? Can you paste an example ? Do you, then, get a single "completion" as an stdio result and ... you can just get new ones by up-arrowing and repeating your command ? Am I getting warm here or am I stuck in a unix command centric paradigm of what this all looks like ?
- nullc 6y ago
  More or less. I have tools I wrote that takes sampling settings and a string and dispatches it across a cluster of machines. Via cluster-ssh I see the each of them expanding the text in a different random sampling. This is custom stuff I've wrote that is tied to my environment. If you'd just like to play around, I can recommend https://bellard.org/nncp/gpt2tc.html https://bellard.org/nncp/gpt2tc.html in text generation mode as being extremely easy to get going. (the default model however is pretty small and dumb). The paradigm you're thinking about is exactly what you get from gpt2tc. At any point I can abort a job, tweak the sampling settings, or the text I'm expanding. E.g. one operation is that if I see one sample seems to be on a good path but has made an error, I'll abort it and restart all of them from a fixed version of that sample. Often I'll end up with my prompts in text files because they get a bit long at times, also escaping quotes and linebreaks on the commandline can be a pita. I have some aspirations of integrating this into a text editor, so that as I type future text is just appearing ahead of me and I can just hit a cursor to accept parts of it. But in my experience GPT2 isn't good enough where looking only at one continuation is enough or where I don't get a lot of advantage in having it work from modified text. GPT2 has preconceived notions about what kind of text you're writing based on the words you use. So it can be useful to alter your input text to replace persons/places/things names with different ones that get in into the right context and then back substitute them. To give a concrete example, if I wanted GPT2 to show me example bio blurbs for my partner (always a pain to write but easier if someone generates examples), it works better if I change her name--Kat-- because it either turns her into a man or it resists talking about her being a lawyer and a board member and instead makes her into an artist or a dancer. One thing to watch out for is that when GPT2 makes a benign error, like switching the gender of a pronoun mid-stream it often trashes the quality of the later output in unexpected ways (like causing it to output nonsense). Changing my SO's name to something it's not unsure about saves me time having to abort completions that have gone off the rails. Perhaps I shouldn't use a gendered example. GPT2 isn't sexist its everything-ist. Every word has 1001 hidden meanings that subtly bias its behaviour, many of the biases are actually the point- they exist in the world and they're what makes the output useful-- others are weird and unexpected and are just a training/corpus artifact. Good use requires a degree of anticipating and exploiting these biases. I even have to change my own name, because GPT2 knows very well that "Greg Maxwell" has something to do with Bitcoin and it readily lapses into Bitcoin conspiracy theories if my name is used. ["Gessh, even the machines are harassing me!"] Another class of biases is language that trips it into fantasy land or 'silly' writing. Any word that is commonly used in writing for children is at risk. If there is another common word that only adults use it might be a better choice if you want serious text, or you setup a context that makes the meaning more clear. It looks like for GPT3 you don't have to use subtle hints as much. E.g. that you can just tell it more explicitly what kind of thing you're doing and won't get tripped up as much by spurious correlations. OTOH, since it doesn't look like they're going to release the GPT3 model it's likely that I'll never get an opportunity to use it in my workflow.