4 ms·

If you want to echo the last paragraph of the article, the system also doesn’t know when to stop. Which leads to a simple improvement: teach the model when to

by bertil 5y ago

If you want to echo the last paragraph of the article, the system also doesn’t know when to stop. Which leads to a simple improvement: teach the model when to stop, what sections to remove. There’s already great ML-based summary engines, and systems able to answer questions from descriptions.

Smithalicious 5y ago
I wonder if that could be a viable approach, having a separate "writer" and "editor" AI
- jcims 5y ago
  I do the same thing when I write (except for my comments on HN lol) GPT-3 reads like the unfiltered garbage that comes out of my head when I'm trying to get ideas out. The primary difference is that I have empathy for the audience so I go through the effort to reread, clean it up, clear it up then cut it down. But that secondary editing process has nothing to do with the subject matter or initial content generation process. It was trained through years of *not* doing it and having my audience ask lots of questions and clearly misunderstand what I was trying to say (let's call it reinforcement learning). As a result I now have an intuition about which concepts need explanatory analogies, based on my understanding of what the readers know, and which are almost certainly going to be understood outright. I think this gets at the root of the difference. Basically when I write, I'm not trying to generate text, I'm trying to generate training data that will amend foreign models with a high fidelity representation of a subset of mine. This feels GAN-like but different. It's like you need to have two separate models, the 'expert' and the 'n00b' (or ten n00bs trained on different data), ask both of them the same thing, and one-shot retrain the n00b from the answers provided by the 'expert' until the n00b is able to generate content that the expert finds to be sufficiently similar.
  - isaacimagine 5y ago
    > Basically when I write, I'm not trying to generate text, I'm trying to generate training data that will amend foreign models with a high fidelity representation of a subset of mine. Well said! Replying so I can find this thread again, please ignore.
    - btrettel 5y ago
      In the future, rather than cluttering up the thread, bookmark the page in your browser or use the "favorite" button here on HN.
- syrrim 5y ago
  GPT-3 is the editor. The hard part of writing is getting a decent idea onto the page. GPT-3 can't do that, it doesn't think for itself. It is good at the second part of writing, creating a coherent string of sentences describing an idea. But since GPT-3 doesn't have decent ideas, the final product doesn't end up being interesting either. The most interesting use cases for GPT-3 involve pairing it with a human who will contribute the actual thinking to the project, with GPT being there to come up with interesting turns of phrase and nice conjoining sentences. Having an AI in the driver seat is still a long way off.
NoOneNew 5y ago
Yea, but that misses the point to all the qualms against the idea that this is "good writing". I know the name of this eassy structure changes depending on region and generation, but 3.5 essay structures teaches you the basic principles of "good writing". What are you arguing, quickly overview your 3 points to prove it, next 3 paragraphs explain and support each point, then a conclusion paragraph. Super simple, basic and on the nose writing to help teach kids how to write WITH PURPOSE. ML based writing doesn't even come close to this. It's just good at the statistical chance of what words follow each other. It's a keyword hound that has some notion of what makes a sentence "look real". People who are afraid of the current generation of ML writing are absolutely dogshit writers with terrible taste in writing. I keep forgetting who said this, but theres a saying to never start writing by describing the weather. Lots of folks miss the point to this. You could modernize it by saying, "Dont start talking about your childhood when writing a recipe." Why? Because no one fucking cares. They want the recipe. Not a description of your childhood or what the weather was like. Take the time to notice articles and "journalist writing". If it starts with the weather, notice how incredibly long it takes for the author to get to the point or describe anything functionally meaningful. You'll notice these articles are extremely long in word count and extremely low in actual information/content. These are the people afraid because yes, a machine can keyword drop and fluff better than them. Is it possible that there will be a future AI/ML to write with purpose? Maybe. But realistic idea connection and logical flow is still a huge hurdle for AI/ML. Summary engines seem good on the surface, but again, if you actually read them for a real world reason, not for "look how good ML is", you'll notice they're as meaningful as a politician running for office. Lots of keywords, lots of nothing really said or done... just like anyone with a liberal arts degree.
- alvah 5y ago
  Correct me if I'm wrong, but I suspect you misunderstand the purpose of all those recipes with ~2000 words of ramble followed by ~300 words of actual recipe. The 2000 words of fluff isn't for the reader, it's 100% for the search engine. It's required to provide enough room to crowbar in the required keywords without triggering an over-optimisation penalty. All in the name of getting it ranked in the SERPs, so people will actually read the ~300 words, or at least click on an ad or two. Unfortunately this is where search engine algorithms have taken us, for now at least.

by bertil 5y ago

Smithalicious 5y ago

I wonder if that could be a viable approach, having a separate "writer" and "editor" AI

jcims 5y ago
I do the same thing when I write (except for my comments on HN lol) GPT-3 reads like the unfiltered garbage that comes out of my head when I'm trying to get ideas out. The primary difference is that I have empathy for the audience so I go through the effort to reread, clean it up, clear it up then cut it down. But that secondary editing process has nothing to do with the subject matter or initial content generation process. It was trained through years of *not* doing it and having my audience ask lots of questions and clearly misunderstand what I was trying to say (let's call it reinforcement learning). As a result I now have an intuition about which concepts need explanatory analogies, based on my understanding of what the readers know, and which are almost certainly going to be understood outright. I think this gets at the root of the difference. Basically when I write, I'm not trying to generate text, I'm trying to generate training data that will amend foreign models with a high fidelity representation of a subset of mine. This feels GAN-like but different. It's like you need to have two separate models, the 'expert' and the 'n00b' (or ten n00bs trained on different data), ask both of them the same thing, and one-shot retrain the n00b from the answers provided by the 'expert' until the n00b is able to generate content that the expert finds to be sufficiently similar.
- isaacimagine 5y ago
  > Basically when I write, I'm not trying to generate text, I'm trying to generate training data that will amend foreign models with a high fidelity representation of a subset of mine. Well said! Replying so I can find this thread again, please ignore.
  - btrettel 5y ago
    In the future, rather than cluttering up the thread, bookmark the page in your browser or use the "favorite" button here on HN.
syrrim 5y ago
GPT-3 is the editor. The hard part of writing is getting a decent idea onto the page. GPT-3 can't do that, it doesn't think for itself. It is good at the second part of writing, creating a coherent string of sentences describing an idea. But since GPT-3 doesn't have decent ideas, the final product doesn't end up being interesting either. The most interesting use cases for GPT-3 involve pairing it with a human who will contribute the actual thinking to the project, with GPT being there to come up with interesting turns of phrase and nice conjoining sentences. Having an AI in the driver seat is still a long way off.

NoOneNew 5y ago

Yea, but that misses the point to all the qualms against the idea that this is "good writing". I know the name of this eassy structure changes depending on region and generation, but 3.5 essay structures teaches you the basic principles of "good writing". What are you arguing, quickly overview your 3 points to prove it, next 3 paragraphs explain and support each point, then a conclusion paragraph. Super simple, basic and on the nose writing to help teach kids how to write WITH PURPOSE. ML based writing doesn't even come close to this. It's just good at the statistical chance of what words follow each other. It's a keyword hound that has some notion of what makes a sentence "look real". People who are afraid of the current generation of ML writing are absolutely dogshit writers with terrible taste in writing. I keep forgetting who said this, but theres a saying to never start writing by describing the weather. Lots of folks miss the point to this. You could modernize it by saying, "Dont start talking about your childhood when writing a recipe." Why? Because no one fucking cares. They want the recipe. Not a description of your childhood or what the weather was like. Take the time to notice articles and "journalist writing". If it starts with the weather, notice how incredibly long it takes for the author to get to the point or describe anything functionally meaningful. You'll notice these articles are extremely long in word count and extremely low in actual information/content. These are the people afraid because yes, a machine can keyword drop and fluff better than them. Is it possible that there will be a future AI/ML to write with purpose? Maybe. But realistic idea connection and logical flow is still a huge hurdle for AI/ML. Summary engines seem good on the surface, but again, if you actually read them for a real world reason, not for "look how good ML is", you'll notice they're as meaningful as a politician running for office. Lots of keywords, lots of nothing really said or done... just like anyone with a liberal arts degree.

alvah 5y ago
Correct me if I'm wrong, but I suspect you misunderstand the purpose of all those recipes with ~2000 words of ramble followed by ~300 words of actual recipe. The 2000 words of fluff isn't for the reader, it's 100% for the search engine. It's required to provide enough room to crowbar in the required keywords without triggering an over-optimisation penalty. All in the name of getting it ranked in the SERPs, so people will actually read the ~300 words, or at least click on an ad or two. Unfortunately this is where search engine algorithms have taken us, for now at least.