15 ms·

The computers are getting better at writing

154 points by jgalt212 6y ago

enkid 6y ago
Is it just me, or is the "continuation" grammatical but completely nonsensical? It talks about infants being thrown into dust bins and calling them animals - not in an intentional way. It talks about trying to crawl away from the blood on his belly. It talks about bugs being ill. It just doesn't actually make sense.
- Swizec 6y ago
  It makes sense the same way a horoscope or a Myers-Briggs test makes sense — the reader adds their own meaning.
- malloryerik 6y ago
  Absolutely, because GPT-3 is a super hot rod text prediction algorithm but has strictly zero consciousness or concept of what it's writing about. It's merely using statistics to do more complex versions of stuff like, when words a, b, c, d, and e are used together, words f, g, h, and j are typically used as well. Take this simple kind of idea and push it to the current max et voilà, GPT-3.
- dash2 6y ago
  I'm not sure it makes much less sense than Kafka, which starts with the hero waking up as a cockroach.
lbacaj 6y ago
I have been doing a lot of writing lately, short and longer essays. I have also written on-and-off for the last ten years. Writing well is not easy. I used to think I wrote well until I started reading better writers. The whole idea that we will just slap together a bunch of text using some “AI” and it will resonate with people is crazy to me. As an engineer whose dabbled with some AI and ML, and has even seen what GPT3 can do, we are as far away from being able to use AI to write a best seller as we are from landing on the Sun. Not saying it will never happen just that nothing we are building right now will be able to do that. Good writing factors in human culture, sparks emotions in people, tells stories that resonate. I will keep going to sleep at night safely assuming that no AI will out-write humans anytime soon and you should too.
- spoonjim 6y ago
  Computer generated text have tens of millions of people they can put out of a job, manipulate into emotions, phish for scams, etc. without needing to produce text like James Joyce.
  - nautilus12 6y ago
    I almost feel like the obscure to most effects of joyce may be easier for gpt3 to generate than other things an only joycian scholars of which there are few would be able to tell
- Swizec 6y ago
  You are right. AI will not replace good writing. AI will replace typical writing – seo spam, news reports, ads, coding tutorials, etc Good writing has insight. AI can’t do that yet. Most writing is filler lettuce. AI is great.
  - ryandrake 6y ago
    Most news articles don’t seem to be even proofread anymore! At least AI spells perfectly. It will at least likely soon replace the uncreative who/what/where/when news writing.
    - tsimionescu 6y ago
      Only for papers that don't care about who/what/where/when, because GPT-3 itself doesn't really care about those questions.
    - YeGoblynQueenne 6y ago
      But if you write such an article, you need to convery specific information - about the "who/what/where/when" of the news story. This is information that was not available to GPT-3 at the time of training (e.g. it doesn't know that Biden is the PotUS now) so someone will have to sit down and craft a prompt that contains this information and such that it causes GPT-3 to generate a piece that repeats this information. To do all that sounds like a lot more work than just writing the article by hand.
causality0 6y ago
GPT-3-generated writing is a combination of comically absurd and incredibly deep. The weak infants of the woman next door who had to be picked up with the tongs and thrown into the dustbin is quite silly but then there's He wanted to crawl away from it, but there was no place to go. which is a surprisingly deep statement about the revulsion he feels towards himself. That a computer can generate that says very uncomfortable things about the skill of humans in recognizing genuine creative significance.
- throwawaygh 6y ago
  > which is a surprisingly deep statement People used to say the same thing about Markov chains, and then the hype died down it took 20 years for another hypetrain to pull into the station. The same will happen with GPT-n. > says very uncomfortable things about the skill of humans in recognizing genuine creative significance. Indeed. People see pictures in clouds and omnipotent beings in the stars. Humans are uncomfortably good at finding meaning in nothings.
  - cageface 6y ago
    What applications are Markov chains used in now?
    - amarant 6y ago
      Kubernetes uses them to come up with random generated cluster names if you don't provide a name of your own..I bet many other applications use them in a similar manner
    - FractalHQ 6y ago
      Crypto currencies and decentralized protocols like the libp2p (used in IPFS) like Markov chains iirc
    - Mordisquitos 6y ago
      Sequence analysis in Bioinformatics, for example to align sequences and detect homology and as in HMMER: http://hmmer.org/ http://hmmer.org/
vumgl 6y ago
Computer-played chess or go games are already more interesting than the human-played ones. If you get computer-written novels indistinguishable from the best human-written novels, or even better, would you read them? There will be 1 million of them generated every day, all better than Harry Potter, or by taste, War and Peace.
- causality0 6y ago
  The better question is, would you read anything else? What happens when we can just dial up the story we want? What happens when these models combine the skillfulness of a human writer with the command of facts, theories, behaviors, and lore of computer records? Imagine. You could feed it all of Tolkien's writings and tell it to give you an epic saga of novels set in the First Age of Middle Earth. You could tweak a parameter and instantly get a version of your favorite novel that proceeds with a different main character or a different choice and have it be indistinguishable from a version written by the original author and one hundred percent consistent with events and previous plots. I'm not too ashamed to say that if memory-erasure technology existed I'd be sorely tempted to experience my favorite media for the first time over and other. Creative AI is the looming spectre of that danger, as it becomes more and more skilled at giving us the best, we will grow intolerant of anything less.
  - Digit-Al 6y ago
    Don't you think there's a danger you'd end up with stories that looked like JRR Tolkien as written by George RR Martin?
    - causality0 6y ago
      I mean, maybe if I was running GPT-3 on a Raspberry Pi. https://www.youtube.com/watch?v=oShTJ90fC34 https://www.youtube.com/watch?v=oShTJ90fC34
  - YeGoblynQueenne 6y ago
    >> Imagine. You could feed it all of Tolkien's writings and tell it to give you an epic saga of novels set in the First Age of Middle Earth. You could tweak a parameter and instantly get a version of your favorite novel that proceeds with a different main character or a different choice and have it be indistinguishable from a version written by the original author and one hundred percent consistent with events and previous plots. Arguably, you are describing about 80% of the Fantasy genre, only it's all written by humans. Sorry if that sounded like a "shallow dismissal" as per the HN guidelines. I used to love F/SF and then I found I couldn't read it anymore because it all felt like the same few story elements were reused over and Over and OVER again. I get the same feeling of dread boredom when I browse the Fantasy isle at bookshops today. <evil power> is rising in the <cardinal direction>. <Hero> must <undertake heroic quest> to defeat <evil power>. It all got so formulaic you could write a Cluedo variant based on it: The Orc Captain with the Great Axe in the Dark Tower. This, btw, was all before I (very belatedly, because I'm not a native English speark and so I read only what was translated to Greek) discovered Terry Pratchett, Jeff Noon and Iain M. Banks, who were a breath of fresh air (unfortunately rarely refreshed since then) and yeah, as a kid, I read a lot of F/SF. Explains my school grades, hah.
nickpinkston 6y ago
Maybe one day computers will even learn to write like the New Yorker... starting every article an initial page of irrelevant artful content and then moving to an overly flowery essay lacking actual insights into the topic at hand.
- witherk 6y ago
  sheeeeeeesh
  - sjg007 6y ago
    everyone's a critic
- dang 6y ago
  "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - nickpinkston 6y ago
    I wouldn't say it's shallow. I think the New Yorker's editorial board / style guide does a disservice in explaining science / engineering concepts. I just delivered my critique via sarcasm.
    - dang 6y ago
      Since it didn't go into anything specific or interesting, I'm afraid it counts as shallow—especially because it's also a cliché response to NYer articles.
      - nickpinkston 6y ago
        Haha - cliché or common knowledge?
jvanderbot 6y ago
This all reminds me of the poetry and intentionally over-used analogies and similes of junior high. People who look for meaning in it will find it, but it's mostly nonsense. It reminds me of the early face generation ... The eyes nose etc were all in the right place, but it was mimicking the appearance not the structure. I'm more interested in a convincing paragraph than sentence.
rdedev 6y ago
I wish GPT 3 api was public. I tried out GPT 2 when I was writing out my SOP but it was really bad whenever there was a pretty technical word in the sentence. I really want to see how GPT 3 handles those cases
TheDudeMan 6y ago
"teraflops per second"
- Clewza313 6y ago
  Teraflops per second is the speed at which computing speeds up. Eyeballing Wikipedia's list of fastest supercomputers, between 2018 and 2020 we clocked around 100 petaflops per year, which comes to around 0.00317 teraflops per second. For homework, you can calculate teraflops per second per second, which would measure the acceleration of teraflops.
aimor 6y ago
What level of Fully Self Writing are we at right now? Maybe L1 with L2 on the horizon? I like the idea of text generation as a writing aid, even a flawed network can help a stuck brain. But say writers really do hate writing, I still come back to the big question: Who's going to read it all? Cheap writing is bad enough already (think stock market news, recipe blogs, or any of my HN comments), I can't begin to fathom how much text I'll be scrolling through once GPT-7 is released. I suppose the automated writing will be consumed by automated readers then, and my involvement starts to feel kind of unnecessary.
wsh 6y ago
I had to rewrite a letter for someone the other day and used Google Docs for the first time in a while. I was struck by how often it made autocomplete suggestions and how good they were: not just stock phrases (“Thank you in advance for your prompt attention...”) but substantive text, too. This kind of help with the mechanics of composition and usage, however, is a far cry from the critical thinking needed to be an effective advocate, expositor, or entertainer. There’s more to being a writer than making plausible sentences and paragraphs.
amilios 6y ago
It's clear that the system occasionally struggles in making coherent sentences that communicate actual meaning -- others have pointed this out well enough. I think what's occasionally lost in these types of criticisms of these systems is the point the article is making about the transformation of the role of the writer from a producer to a curator. Already this transformation has occured in other fields; the job of a professional translator (for text) at this point is mostly starting with a machine translation and slightly tweaking it in order to produce something more natural. I can easily see this happening with GPT-N: the writer inputs some ideas, and filters what's produced and/or tweaks it to make the final product. It doesn't matter if X% of what it produces is trash: if there is usable content, the role of the writer has been changed. If the machine produces something like the article's examples on the first go, and all the "writer" needs to do is tweak it a bit to modify the bit about throwing babies in the trash, then it is still groundbreaking.
- groby_b 6y ago
  > the job of a professional translator (for text) at this point is mostly starting with a machine translation and slightly tweaking it in order to produce something more natural. Not really, no. That might be true for instruction booklet translation (they sure read like it), but no, for a decent translation, you'll need to let a human have a full go at it. Writing is as much about filtering & tweaking GPT-3 output as sculpting is about "removing everything that doesn't look like the final sculpture", as the old adage goes. It is useful for one purpose: To do an absolutely minimal job and to externalize some more work onto end users. It'll also be popular for spam^W SEO, and for writing filler content for content graveyards. It will not, however, be useful for writing. I have no doubt that we'll see a lot more tooling supporting writing, and machine-assisted writing. I have equally no doubt that GPT-3 and friends aren't going to be it.
  - Clewza313 6y ago
    Sure, "decent" translation as in novels etc will continue to be hand-crafted, but the vast majority of translation is low-end bulk stuff like business reports and technical documentation, paid pennies per word. I know one (former) professional translator, and dumping text into Google Translate was always her starting point. Interestingly, her job got harder when Google rolled out the new deep learning models, because instead of outputting visibly broken garbage that kind of flagged itself for checking, it now tends to produce beautifully formed sentences with subtle but often critical fuckups (wrong sense of a word, missing negation, parsing a pronoun incorrectly, etc) that can affect meaning.
minimaxir 6y ago
See also my experiments with GPT-3 on sane prompts, which have wildly varying quality even after generating them in bulk: https://github.com/minimaxir/gpt-3-experiments https://github.com/minimaxir/gpt-3-experiments Creative writing hasn't been one of the super-hyped use cases by OpenAI for the OpenAI API outside of AI Dungeon, surprisingly. For just random generation, the necessary curation can detract from the time-savings advantages. (as an aside, the API is also extremely expensive for long-form content to the point I'm not sure how the economics work for these startups even with charging monthly fees). I'm more bullish on small bespoke models for a given use case, which is what I spend my time researching. (example: a ~1M parameter GPT-2 Magic: the Gathering card generator: https://colab.research.google.com/drive/1VOt090UzvltoBgMdUZmU5vwhi4X-6E_a?usp=sharing https://colab.research.google.com/drive/1VOt090UzvltoBgMdUZm... )
- YeGoblynQueenne 6y ago
  Generating M:tG cards is interesting to me. Can we talk about this a bit? :) My degree dissertation was an M:tG expert system that included a hand-crafted parser and generator for (a small subset of) ability text [1]. My Masters' thesis was a grammar induction system trained on M:tG cards [2]. And I finally managed to sneak some M:tG grammar induction in my papers as a PhD student [3]. I had a quick look at the colab of your project. If I may be critical, the examples you have are ... so-so. I've seen previous attempts to generate M:tG text with neural nets, starting with the old project in M:tG Salvation that became Roborosewater and that used LSTMs if memory serves. Results have always been a mixed bag that looks promising but always falls down holes even with hand-curation. This still seems to be the case with your project. A couple of examples (you probably know already): −2: Put a +1/+1 counter on each other creature you control. "Each other creature" should refer to a card with the (super) type "creature" but this card is a "planeswaker". +1: Each opponent chooses a creature they control. If that creature was a copy of that creature, each other player reveals their hand, chooses one of those cards, then puts it into their hand, then shuffles. Which creature was that creature a copy of? −10: Target opponent sacrifices a creature. −9: Each opponent sacrifices a creature. Representing costs in a meaningful manner is a constant problem in every M:tG generator I've seen. The problems I highlight above are not with grammaticality, which is certainly a big step forward with respect to the past. But many of the abilities still don't make a lot of sense, or don't make sense to be on the same card, or have weird costs etc. If you want your project to be something genuinely new, I have to say I don't think it's there yet. My intuition is that it would take a lot more than language modelling to generate M:tG cards that make enough sense that it's more fun to generate them than create them yourself. I think it would be necessary to have background knowledge of the game, at least its rules, if not some concept of a metagame. It might also help to re-train standard NLP pipelines specifically for M:tG (e.g. some kind of ability-text specific anaphora resolution might help with "that creature" above). Also, I note that the new online version of the game is capable of parsing cads as scripts in a programming language using a hand-crafted grammar rather than a machine-learned model [4] [5]. So it seems to me that the state-of-the-art for M:tG language modelling is still a hand-crafted grammar. __________________ [1] https://github.com/stassa/Gleemin https://github.com/stassa/Gleemin - unfortunately, doesn't run anymore after multiple changes to Prolog interepreters used to create and then port the project over. [2] https://github.com/stassa/THELEMA https://github.com/stassa/THELEMA - should work with older versions of Swi-Prolog, unfortunately not documented in the README. [3] https://link.springer.com/article/10.1007/s10994-020-05945-w https://link.springer.com/article/10.1007/s10994-020-05945-w - see Section 3.3 "Experiment 3: M:tG fragment". Cringe-inducing to read just a few months after writing, sorry. [4] https://www.reddit.com/r/magicTCG/comments/74hw1z/magic_arenas_gre_the_groundwork_for_future/do0rvfe/ https://www.reddit.com/r/magicTCG/comments/74hw1z/magic_aren... [5] https://www.reddit.com/r/magicTCG/comments/9kxid9/mtgadispersal_bugged/e73pr9u/ https://www.reddit.com/r/magicTCG/comments/9kxid9/mtgadisper...
notum 6y ago
I'll admit, I just scrolled to the bottom to see if the article ends with "This piece has been written by GPT-3". It doesn't.
Animats 6y ago
GPT-3 is already better than most blog comments, many bloggers, and most online text porn. Maybe most ad copy. Many pundits. Some college professors. It has no clue what it's talking about, of course. It's terrible at fact-based material, because what comes out looks plausible, but contains made-up facts. Some facts will be correct, because they were mentioned a few times in the training database. Others are just plausible words strung together. What it really demonstrates is that much of literature has less behind it than was previously supposed.
- FeepingCreature 6y ago
  I expect GPT-3 to be amazing for people who want to write stories or articles but are really terrible at one specific aspect of the writing process.
  - sbierwagen 6y ago
    https://slideplayer.com/slide/6081467/18/images/12/The+scale+of+intelligent+minds%3A+a+parochial+view..jpg https://slideplayer.com/slide/6081467/18/images/12/The+scale... It depends on if you think writing ability is more like the top graph, or the bottom graph. If it's like the top graph, humans have plenty of time to adopt AI tools, and incorporate them into the writing process. If it's like the bottom graph, then GPT-4 comes out in 12 months and is human equivalent, and GPT-5 comes out a year after that and is a better writer than any human could hope to be.
    - Animats 6y ago
      Progress so far is that GPT-1 could produce a coherent sentence but stopped making sense after a few lines. GPT-2 could produce a coherent paragraph, but several paragraphs strung together made no sense. GPT-3 is coherent for about a page or screen, but not a chapter. A page is good enough for the clickbait industry.
- bpodgursky 6y ago
  I have a friend who uses GPT-3 to generate startup marketing copy and customer-facing emails. He just prompts with the company description and "The email which is most effective at getting customers to sign up for <startup> is:" It reads as good or better than a human would do. Maybe not better than the BEST human marketer would do, but far better than average, and instant + free(ish).
mattikl 6y ago
Rewinding a bit from computers doing all the writing, are you using any tools that help you write considerably better (not just spelling/punctuation checking)? As a non-native English speaker I would like to have a tool that points out my nonstandard usages of language. I think it could be something as simple as parsing the text and searching it against a corpus, but I have no idea how many "unique structures" a text typically contains where no one else has used those same exact words. AI could help to suggest better ways to formulate the text.
- camillomiller 6y ago
  As a fellow non-native who accepted not to strive for full assimilation, my suggestion is to do the same. You can refine your writing by reading a lot. Books like “The elements of style” and “On writing well” can help with grasping those structural elements you might be struggling to identify. Apart from that, our brain has been shaped by our native language, and we can’t really delete that. Which is great news! It means we can express concepts in English with structures and flows that, if grammatically correct, have the advantage of sounding original, different, sometimes exotic. One tool I use is Grammarly. I think it’s mostly crap, but if used properly it helps you identify the more blatant mistakes. Oh, and by the way, if you need a non-native speaker ego booster, try and read native speakers’ first drafts of anything.
  - mattikl 6y ago
    That's a good point of turning your native language into an asset. The goal of communication is to be well-understood. That's so easy to forget that and strive for something more. Good writing also keeps the readers engaged, maybe entertained, but is there much more to that?
  - thaumasiotes 6y ago
    > It means we can express concepts in English with structures and flows that, if grammatically correct, have the advantage of sounding original, different, sometimes exotic. Yeah, that happens all the time. It always strikes me as unfair when someone comes out with an ostensibly perfect sentence and the result is "well, you didn't make any mistakes; all the words mean what you want them to mean and all the grammar is used correctly. But the sentence is still wrong; that's just not how we say that."
8bitsrule 6y ago
Couldn't help but notice that 'Xanadu' uses a lot of rhymes ... GPT apparently didn't notice that constraint. And then there's: "While all the stars that round her burn’d, Bow’d to the ground and based their fires. To the one ever-branching cloud" When 'base' is used as a past-tense verb, we expect a prepositional phrase ... before the next period. The preposition is never 'to'. Doesn't matter; the period stopped the English part. "That blew and drifted—blow and drift;" What? "If you told me that Coleridge wrote it, I would believe you." NExt.
- gwern 6y ago
  > Couldn't help but notice that 'Xanadu' uses a lot of rhymes ... GPT apparently didn't notice that constraint. Due to an obscure low-level technical decision (encoding text into BPEs), phonetics are erased from the GPT training data, and so therefore neither GPT-2 nor GPT-3 can learn to rhyme or make puns (they can only memorize relatively common pairs by bruteforce): https://www.gwern.net/GPT-3#bpes https://www.gwern.net/GPT-3#bpes I put a good deal of effort into testing this, including "Kubla Khan": https://www.gwern.net/GPT-3#samuel-taylor-coleridge https://www.gwern.net/GPT-3#samuel-taylor-coleridge Just doesn't work, and it never will until OA trains a model using some sort of better encoding which preserves phonetic information. (Character-based, unigram LM, or BPE dropout might all work, we don't know yet.)
caconym_ 6y ago
The lesson we should take from GPT-3 is that we spend way too much time reading text that isn't really saying anything, and/or failing to find meaning in what we read. It's a neat trick, and I think it could absolutely replace much or most of the writing on the internet today. But is it good writing? No, not really. At least, not in the sense of writing as producing anything more substantive than slick prose with a vague, meandering context running through it. I think the next step (say, being able to produce short stories with actual literary value) will be substantially more difficult, mostly because it's substantially more difficult for human writers too. It takes learned abstract knowledge and deliberate intention to write something that's coherent and engaging and longer than a few hundred words, and that skill is separate from the intuitive prose generation ability that's essentially unconscious for practiced writers and comes easily to many avid readers (i.e. trained on a large corpus) who take up writing. It will be interesting to see how producers of actual substantive content incorporate these tools into their processes, if at all. I guess I can imagine using it to generate descriptions, for instance, but how does it do with dialogue? Probably not very well.
- DyslexicAtheist 6y ago
  > But is it good writing? No, not really. agree. I'd say it's impossible to solve. Where would we even place our yardsticks: Has society been able to get any better (than we used to) at assessing and categorizing what "constitutes" great writing? If so in what relation could that be measured to our past performance especially that many no longer read today due to our loss of attention span? (most won't even read this comment because it's way too long). People are terrible at spotting some of the best writing when it happens. History is full of writers who's work could only be understood and appreciated after their death.[1] Death makes their work a finite commodity. There is no more chance asking the author "what they meant". Others get shunned because they were too radical, (or too ahead of their times). Great writers as in "a radical new style" or as "for their radical ideas" get only credited after decades (if not centuries) post mortem. When it's too upsetting (because it's true) we need the distance of history and later generations to appreciate it[2]. Even "what" we comprehend and then appreciate changes over time, and within the same person with every time we read it[3]. [1] 6 people centuries ahead of their time https://www.oxford-royale.com/articles/6-people-centuries-ahead-time/ https://www.oxford-royale.com/articles/6-people-centuries-ah... <- there are hundreds of wrtiers like this (it is just the top result google gave me). How would an AI spot it when all AI was created "in our image". And if it did spot it how would the AI be able to convince us that this is worth a closer look instead that we need to destroy the AI? [2] The Last Messiah by Peter Wessel Zapffe https://philosophynow.org/issues/45/The_Last_Messiah https://philosophynow.org/issues/45/The_Last_Messiah <- say something mediocre to ensure you get published and ranked in "best new books". But saying something profound can potentially get you ignored (at best) and killed (at worst). This ties back to[1] [3] James Joyce Dubliners https://archive.org/details/dubliners00joycrich https://archive.org/details/dubliners00joycrich / https://en.wikipedia.org/wiki/Dubliners https://en.wikipedia.org/wiki/Dubliners <- there are hundreds of examples like Dubliners but they are all different books depending on who reads them and when. the retrospect and integrating what we read (and then re-read over and over at various stages of our lives) into our own reality is what is makes it great. yet even if some minority of people does practice this, it's still highly subjective. how would an AI provide us with value (e.g. if it can the value it creates would only benefit itself but never others) __ I'm not saying AI is incapable of better writing than AI is now, but it would only remain mediocre and never outperform humans. Also the writing can't stand or be judged by itself. It can only be valued and appreciated through human mind. Art (and many books are art) has the same problematic relationship with AI. John Berger's "Ways of seeing" talks about this (but so does a lot of philosophy) https://www.youtube.com/watch?v=0pDE4VX_9Kk https://www.youtube.com/watch?v=0pDE4VX_9Kk
cjohansson 6y ago
The AI generated paragraph made no sense to me and did not sound like the first. AI is like a narccisist, every other day people say it's useless and then the next it's the savior of the world.
ayushchat 6y ago
I've actually used tools like copy.ai and converstion.ai for a couple of copywriting projects and found them to be very useful. Of course the output isn't perfect, human intervention is required, but it is way better than doing the writing myself from scratch. I think going forward we are going to see a lot of human + AI writing projects instead of purely human or purely AI.
- nxpnsv 6y ago
  * you mean https://www.conversation.ai/ https://www.conversation.ai/ ?
  - dave84 6y ago
    I'd imagine it's https://www.conversion.ai/ https://www.conversion.ai/
    - nxpnsv 6y ago
      Yes, looks better... thank you!
  - ayushchat 6y ago
    yes right, typo I meant https://www.conversion.ai/ https://www.conversion.ai/ Their AI assistant is called Jarvis
db39 6y ago
This is something I've been deep into recently. Just a couple of days ago I published a short paperback story (grimtalesbook.com) written and illustrated by AI - GPT-3 for the writing in this case. I liked the analogy aimor made in this thread between the level readiness of autonomous driving and autonomous writing. From my experience that feels right, we're not at L5 yet - you can't just let the AI write continuously because you'll just end up with junk wrapped around some nuggets of gold. In the future if we get to 'L5' writing, your child will have the ability to say 'OK Google, tell me a story about my adventures as a dragon slayer' and Google will respond happily with a perfect bedtime story. And the next day, you'll get a physical copy of that story in the mail. Luckily, that future isn't here yet. The way Grim Tales was written with GPT-3 was by using some human intervention to keep the AI on track to write a cohesive narrative with a plot and dialogue. A couple of rules like removing anything that doesn't progress the narrative, and keeping the narrator in first person meant that the story was far more readable and didn't meander around like common examples. Right now - for professional writers - AI is probably good enough as a co-pilot used to create alternate scenes at points in the narrative, or as a tool to help with writers' block.
WesolyKubeczek 6y ago
I have a suspicion that, since GPT-3 gorged its enormously swollen fuzzy digital belly on a lot of real prose, it might, when prompted, occasionally throw in a piece of the prose in question, or a whole original passage, because to its fitting function, it would be a suitable thing to do. But then again, aren’t we all doing the same, regurgitating bits and pieces that we heard or read before and liked? Don’t we all sometimes, when busy with our own thoughts and worries, occasionally spout out oddly appropriate responses to what our spouses are speaking, without actually listening to and processing the conversation, maybe because we heard it all so many times we can readily predict what they are going to say? And then we agree that we need to go take the alligator from the repair shop. However, if anything, GPT-3 is mostly showing how little meaning there is, but in how many words. Never mind me, I’ll pour myself another one.
- yesenadam 6y ago
  > Don’t we all sometimes, when busy with our own thoughts and worries, occasionally spout out oddly appropriate responses to what our spouses are speaking, without actually listening to and processing the conversation, maybe because we heard it all so many times we can readily predict what they are going to say? Uh.. no, not me. That sounds very strange to me, and sad. I always listen to everything she says. And expect the same back. Otherwise it wouldn't be a good relationship, I think. (Maybe I'm super-naïve!..or something.)
- claudiawerner 6y ago
  >Don’t we all sometimes, when busy with our own thoughts and worries, occasionally spout out oddly appropriate responses to what our spouses are speaking, without actually listening to and processing the conversation, maybe because we heard it all so many times we can readily predict what they are going to say? No. Sometimes I can predict what a friend will say but I won't actually say it. The more I think about it, the more I realise that AI at the moment is the unlistening (but hearing) spouse. And just like it seems we can engage with such a person (though can only rarely), so too can the AI trick us ('simulate') that behaviour.
- YeGoblynQueenne 6y ago
  >> But then again, aren’t we all doing the same, regurgitating bits and pieces that we heard or read before and liked? We can do that. We can also not do that. Language models can only do that. My comment is stripped of nuance (language models don't do that exactly) but it should point at the difference: humans can be as dumb as bricks, or as smart as humans. Language models don't even come close to our brighter moments. They don't even come close to the brighter moments of our pets, and our pets can't generate human language. You have kids? Think of whom you'd like to spend the next decade or two around: a five year old human or GPT-3? Which one would you find more interesting, original, curious, do you think? I think people who are the most excited about GPT-3 are the ones who are used to even dumber, more boring interactions with computers than the repetitive and unoriginal text generation of language models. Or perhaps it's humans who are used to boring interactions with other humans. Technology has done some weird things to us of late I fear.
bambax 6y ago
Many good and prolific writers of the past used ghost writers who prepared drafts and idea lists for them, in effect making their work more like that of editors (it is said Alexandre Dumas worked like this -- it's one of the only reason to explain his spectacular productivity). So if AI is capable of doing some prep work, why not? When writing fiction, my struggle is usually not in the writing itself (making sentences) but in figuring out what to write about (the content of a scene). An AI that would generate scene ideas from an input of, say, a list of characters and plot points, would be very helpful. Outputting sentences and paragraphs, not so much (I think). Also, the article seems to conflate Sudowrite and GPT-3 but it would be interesting to understand the difference between the two; is Sudowrite just a wrapper over the GPT-3 API that forwards requests directly? If not, what is its added value vs hitting the API directly?
- jamesjyu 6y ago
  Founder of Sudowrite here. We are built on GPT-3 but we have a bunch of things we do on top that optimizes Sudowrite for the narrative fiction/non-fiction use case. For example, we have a description tool that, given a highlighted phrase, gives you options for telling phrases in 5 senses and metaphorical uses. Another tool we have analyzes a story premise and gives you possible twist and story turns, based on the genre of your story, and also gives you possible characters that could inhabit that world, complete with both external and internal character descriptions. I agree that for writing stories, the hard part is in the forest and not the trees (although this isn't true for everyone). And systems like GPT-3 can help with that, too, simply because it's level understanding of text is approaching that of a human.
  - bambax 6y ago
    Interesting! Thank you for answering. How long is the waiting list to join Sudowrite beta?
    - jamesjyu 6y ago
      It’s gotten quite large, but we plan on sending more invites out soon. We are prioritizing folks who are working writers and have an immediate project they’re working on that could benefit from collaborating with an AI. If this sounds like you, sign up on the beta form and also send me a note about how you’ll use Sudowrite to jamesjacobyu@gmail.com
ta_ca 6y ago
until a few minutes ago i was a skeptic. now i think about it... this gpt business is not only AI but it might be a way to AGI. why? it mimics us perfectly that each of us can fill books about the things we have zero understanding of.
jgalt212 6y ago
> In July of 2019, Microsoft invested a billion dollars, which allowed OpenAI to create a supercomputer with two hundred and eighty-five thousand C.P.U. cores, ten thousand G.P.U.s, and four hundred gigabits per second of network connectivity per server. > Gupta imagines the product turning into a resource that writers will pay fifteen to twenty dollars per month to use. That seems like a great deal to sublet some portion of all those resources.
troelsSteegin 6y ago
Are people getting better at reading? A deep read is an immersive, flow experience. But to engage in reading you have feel you have the time and focus to commit to reading and then not be interrupted. I don't want for machine generated narrative, I want more time spent deep reading. I do way more scanning and skimming now than deep reading - authoritative source, credible, what's new, what's useful - it's all critical thinking and done as efficiently as I can make it. It feels like work. I always wonder if more structuring over the stuff that I read - entity and concept extraction, thesis extraction, separating facts from opinions, a topic-centered feed - would be better. I haven't found or built the right experience yet. To me, narrative generating models are interesting only as a step toward a better critical reading experience, and the win there would be generating useful abstractions that help one understand faster. Arguably, writing style is one such abstraction. Maybe there's potential in being able to generate readable narratives from things that people have a hard time describing, like datasets. Then the grail is business dashboards composed as sonnets.
jamesjyu 6y ago
Oh wow, just seeing this post made it to the front page. Founder of Sudowrite here, happy to answer questions.
- libertine 6y ago
  I don't mean to put on you the spot, but I have been wondering about this: Do you think that such tools will have the eyes of the regulators on top of them due to copyright issues, since the data used could have a lot of copyrighted material? I haven't yet made my mind about this: is the output original content, or just shuffled proprietary content?
  - jamesjyu 6y ago
    (Caveat: IANAL) As far as I know the issue of copyright and AI generated content hasn’t been tested in the courts yet. There are two issues: 1) The author inadvertently uses output from GPT-3 that happens to be close enough to copyrighted material to be a problem. This is why we recommend to authors do two things: a) enter as much of their own original work to prompt Sudowrite and b) edit and incorporate the AI output organically into their work. The chance that GPT-3 outputs verbatim copyright work in that case is small. As we grow, we also plan to incorporate plagiarism checks in the background. 2) Someone could challenge the author’s copyright on a piece of work that was primarily written by GPT-3. This has not been tested in the courts, and I suspect it will come to head in the next few years. For me, the question of who owns copyright is dependent on how much work the author has done to the AI output (there are some parallels to the Monkey Selfie case [2]). For example, Stephen Marche (the author of the new yorker post from OP) also wrote a short story [1] using Sudowrite, in which he claims that 17.1% of the final words were from the AI. But of course, he has done non-trivial amounts of work to edit and incorporate those words and made it his own. So in this case, he def deserves ownership. All of our Sudowrite authors use the AI similarly in their process. However, what if an author uses vanilla GPT-3 with no humans int the loop to generate hundreds of biographies of famous people based on wikipedia entries? I don’t have a good answer to this. It shouldmay come down to how much the author has done to guide GPT-3 and/or pre-post-process it. Here’s further reading: http://rutgerslawreview.com/wp-content/uploads/2017/07/Robert-Denicola-Ex-Machina-69-Rutgers-UL-Rev-251-2016.pdf http://rutgerslawreview.com/wp-content/uploads/2017/07/Rober... https://core.ac.uk/download/pdf/77599763.pdf https://core.ac.uk/download/pdf/77599763.pdf [1] https://lareviewofbooks.org/article/the-thing-on-the-phone/ https://lareviewofbooks.org/article/the-thing-on-the-phone/ [2] https://www.npr.org/sections/thetwo-way/2017/09/12/550417823/-animal-rights-advocates-photographer-compromise-over-ownership-of-monkey-selfie https://www.npr.org/sections/thetwo-way/2017/09/12/550417823...
bulldog13 6y ago
What is current state of the art that a user can actually access ? For example, GPT-3 requires open ai beta access. Is it still GPT-2 ?
SpeakerFrThDead 6y ago
Does anybody know what the author means when he asserts that "nearly all writers he knows don't like writing"? Seems contradictory to everything I've heard about the iteration loop between practice, play, and discovery that characterizes mastery in any craft, be it programming, writing, music, sports, and so on.
konfuzio 6y ago
GPT Neo works well for drafting emails. Find our approach here: https://www.helm-nagel.com/open-sourced-gpt-Neo-writes-individualized-emails https://www.helm-nagel.com/open-sourced-gpt-Neo-writes-indiv...

154 points by jgalt212 6y ago

Is it just me, or is the "continuation" grammatical but completely nonsensical? It talks about infants being thrown into dust bins and calling them animals - not in an intentional way. It talks about trying to crawl away from the blood on his belly. It talks about bugs being ill. It just doesn't actually make sense.

Swizec 6y ago
It makes sense the same way a horoscope or a Myers-Briggs test makes sense — the reader adds their own meaning.
malloryerik 6y ago
Absolutely, because GPT-3 is a super hot rod text prediction algorithm but has strictly zero consciousness or concept of what it's writing about. It's merely using statistics to do more complex versions of stuff like, when words a, b, c, d, and e are used together, words f, g, h, and j are typically used as well. Take this simple kind of idea and push it to the current max et voilà, GPT-3.
dash2 6y ago
I'm not sure it makes much less sense than Kafka, which starts with the hero waking up as a cockroach.

lbacaj 6y ago

I have been doing a lot of writing lately, short and longer essays. I have also written on-and-off for the last ten years. Writing well is not easy. I used to think I wrote well until I started reading better writers. The whole idea that we will just slap together a bunch of text using some “AI” and it will resonate with people is crazy to me. As an engineer whose dabbled with some AI and ML, and has even seen what GPT3 can do, we are as far away from being able to use AI to write a best seller as we are from landing on the Sun. Not saying it will never happen just that nothing we are building right now will be able to do that. Good writing factors in human culture, sparks emotions in people, tells stories that resonate. I will keep going to sleep at night safely assuming that no AI will out-write humans anytime soon and you should too.

spoonjim 6y ago
Computer generated text have tens of millions of people they can put out of a job, manipulate into emotions, phish for scams, etc. without needing to produce text like James Joyce.
- nautilus12 6y ago
  I almost feel like the obscure to most effects of joyce may be easier for gpt3 to generate than other things an only joycian scholars of which there are few would be able to tell
Swizec 6y ago
You are right. AI will not replace good writing. AI will replace typical writing – seo spam, news reports, ads, coding tutorials, etc Good writing has insight. AI can’t do that yet. Most writing is filler lettuce. AI is great.
- ryandrake 6y ago
  Most news articles don’t seem to be even proofread anymore! At least AI spells perfectly. It will at least likely soon replace the uncreative who/what/where/when news writing.
  - tsimionescu 6y ago
    Only for papers that don't care about who/what/where/when, because GPT-3 itself doesn't really care about those questions.
  - YeGoblynQueenne 6y ago
    But if you write such an article, you need to convery specific information - about the "who/what/where/when" of the news story. This is information that was not available to GPT-3 at the time of training (e.g. it doesn't know that Biden is the PotUS now) so someone will have to sit down and craft a prompt that contains this information and such that it causes GPT-3 to generate a piece that repeats this information. To do all that sounds like a lot more work than just writing the article by hand.

causality0 6y ago

GPT-3-generated writing is a combination of comically absurd and incredibly deep. The weak infants of the woman next door who had to be picked up with the tongs and thrown into the dustbin is quite silly but then there's He wanted to crawl away from it, but there was no place to go. which is a surprisingly deep statement about the revulsion he feels towards himself. That a computer can generate that says very uncomfortable things about the skill of humans in recognizing genuine creative significance.

throwawaygh 6y ago
> which is a surprisingly deep statement People used to say the same thing about Markov chains, and then the hype died down it took 20 years for another hypetrain to pull into the station. The same will happen with GPT-n. > says very uncomfortable things about the skill of humans in recognizing genuine creative significance. Indeed. People see pictures in clouds and omnipotent beings in the stars. Humans are uncomfortably good at finding meaning in nothings.
- cageface 6y ago
  What applications are Markov chains used in now?
  - amarant 6y ago
    Kubernetes uses them to come up with random generated cluster names if you don't provide a name of your own..I bet many other applications use them in a similar manner
  - FractalHQ 6y ago
    Crypto currencies and decentralized protocols like the libp2p (used in IPFS) like Markov chains iirc
  - Mordisquitos 6y ago
    Sequence analysis in Bioinformatics, for example to align sequences and detect homology and as in HMMER: http://hmmer.org/ http://hmmer.org/

vumgl 6y ago

Computer-played chess or go games are already more interesting than the human-played ones. If you get computer-written novels indistinguishable from the best human-written novels, or even better, would you read them? There will be 1 million of them generated every day, all better than Harry Potter, or by taste, War and Peace.

causality0 6y ago
The better question is, would you read anything else? What happens when we can just dial up the story we want? What happens when these models combine the skillfulness of a human writer with the command of facts, theories, behaviors, and lore of computer records? Imagine. You could feed it all of Tolkien's writings and tell it to give you an epic saga of novels set in the First Age of Middle Earth. You could tweak a parameter and instantly get a version of your favorite novel that proceeds with a different main character or a different choice and have it be indistinguishable from a version written by the original author and one hundred percent consistent with events and previous plots. I'm not too ashamed to say that if memory-erasure technology existed I'd be sorely tempted to experience my favorite media for the first time over and other. Creative AI is the looming spectre of that danger, as it becomes more and more skilled at giving us the best, we will grow intolerant of anything less.
- Digit-Al 6y ago
  Don't you think there's a danger you'd end up with stories that looked like JRR Tolkien as written by George RR Martin?
  - causality0 6y ago
    I mean, maybe if I was running GPT-3 on a Raspberry Pi. https://www.youtube.com/watch?v=oShTJ90fC34 https://www.youtube.com/watch?v=oShTJ90fC34
- YeGoblynQueenne 6y ago
  >> Imagine. You could feed it all of Tolkien's writings and tell it to give you an epic saga of novels set in the First Age of Middle Earth. You could tweak a parameter and instantly get a version of your favorite novel that proceeds with a different main character or a different choice and have it be indistinguishable from a version written by the original author and one hundred percent consistent with events and previous plots. Arguably, you are describing about 80% of the Fantasy genre, only it's all written by humans. Sorry if that sounded like a "shallow dismissal" as per the HN guidelines. I used to love F/SF and then I found I couldn't read it anymore because it all felt like the same few story elements were reused over and Over and OVER again. I get the same feeling of dread boredom when I browse the Fantasy isle at bookshops today. <evil power> is rising in the <cardinal direction>. <Hero> must <undertake heroic quest> to defeat <evil power>. It all got so formulaic you could write a Cluedo variant based on it: The Orc Captain with the Great Axe in the Dark Tower. This, btw, was all before I (very belatedly, because I'm not a native English speark and so I read only what was translated to Greek) discovered Terry Pratchett, Jeff Noon and Iain M. Banks, who were a breath of fresh air (unfortunately rarely refreshed since then) and yeah, as a kid, I read a lot of F/SF. Explains my school grades, hah.

nickpinkston 6y ago

Maybe one day computers will even learn to write like the New Yorker... starting every article an initial page of irrelevant artful content and then moving to an overly flowery essay lacking actual insights into the topic at hand.

witherk 6y ago
sheeeeeeesh
- sjg007 6y ago
  everyone's a critic
dang 6y ago
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- nickpinkston 6y ago
  I wouldn't say it's shallow. I think the New Yorker's editorial board / style guide does a disservice in explaining science / engineering concepts. I just delivered my critique via sarcasm.
  - dang 6y ago
    Since it didn't go into anything specific or interesting, I'm afraid it counts as shallow—especially because it's also a cliché response to NYer articles.
    - nickpinkston 6y ago
      Haha - cliché or common knowledge?

jvanderbot 6y ago

This all reminds me of the poetry and intentionally over-used analogies and similes of junior high. People who look for meaning in it will find it, but it's mostly nonsense. It reminds me of the early face generation ... The eyes nose etc were all in the right place, but it was mimicking the appearance not the structure. I'm more interested in a convincing paragraph than sentence.

rdedev 6y ago

I wish GPT 3 api was public. I tried out GPT 2 when I was writing out my SOP but it was really bad whenever there was a pretty technical word in the sentence. I really want to see how GPT 3 handles those cases

TheDudeMan 6y ago

"teraflops per second"

Clewza313 6y ago
Teraflops per second is the speed at which computing speeds up. Eyeballing Wikipedia's list of fastest supercomputers, between 2018 and 2020 we clocked around 100 petaflops per year, which comes to around 0.00317 teraflops per second. For homework, you can calculate teraflops per second per second, which would measure the acceleration of teraflops.

aimor 6y ago

What level of Fully Self Writing are we at right now? Maybe L1 with L2 on the horizon? I like the idea of text generation as a writing aid, even a flawed network can help a stuck brain. But say writers really do hate writing, I still come back to the big question: Who's going to read it all? Cheap writing is bad enough already (think stock market news, recipe blogs, or any of my HN comments), I can't begin to fathom how much text I'll be scrolling through once GPT-7 is released. I suppose the automated writing will be consumed by automated readers then, and my involvement starts to feel kind of unnecessary.

wsh 6y ago

I had to rewrite a letter for someone the other day and used Google Docs for the first time in a while. I was struck by how often it made autocomplete suggestions and how good they were: not just stock phrases (“Thank you in advance for your prompt attention...”) but substantive text, too. This kind of help with the mechanics of composition and usage, however, is a far cry from the critical thinking needed to be an effective advocate, expositor, or entertainer. There’s more to being a writer than making plausible sentences and paragraphs.

amilios 6y ago

It's clear that the system occasionally struggles in making coherent sentences that communicate actual meaning -- others have pointed this out well enough. I think what's occasionally lost in these types of criticisms of these systems is the point the article is making about the transformation of the role of the writer from a producer to a curator. Already this transformation has occured in other fields; the job of a professional translator (for text) at this point is mostly starting with a machine translation and slightly tweaking it in order to produce something more natural. I can easily see this happening with GPT-N: the writer inputs some ideas, and filters what's produced and/or tweaks it to make the final product. It doesn't matter if X% of what it produces is trash: if there is usable content, the role of the writer has been changed. If the machine produces something like the article's examples on the first go, and all the "writer" needs to do is tweak it a bit to modify the bit about throwing babies in the trash, then it is still groundbreaking.

groby_b 6y ago
> the job of a professional translator (for text) at this point is mostly starting with a machine translation and slightly tweaking it in order to produce something more natural. Not really, no. That might be true for instruction booklet translation (they sure read like it), but no, for a decent translation, you'll need to let a human have a full go at it. Writing is as much about filtering & tweaking GPT-3 output as sculpting is about "removing everything that doesn't look like the final sculpture", as the old adage goes. It is useful for one purpose: To do an absolutely minimal job and to externalize some more work onto end users. It'll also be popular for spam^W SEO, and for writing filler content for content graveyards. It will not, however, be useful for writing. I have no doubt that we'll see a lot more tooling supporting writing, and machine-assisted writing. I have equally no doubt that GPT-3 and friends aren't going to be it.
- Clewza313 6y ago
  Sure, "decent" translation as in novels etc will continue to be hand-crafted, but the vast majority of translation is low-end bulk stuff like business reports and technical documentation, paid pennies per word. I know one (former) professional translator, and dumping text into Google Translate was always her starting point. Interestingly, her job got harder when Google rolled out the new deep learning models, because instead of outputting visibly broken garbage that kind of flagged itself for checking, it now tends to produce beautifully formed sentences with subtle but often critical fuckups (wrong sense of a word, missing negation, parsing a pronoun incorrectly, etc) that can affect meaning.

minimaxir 6y ago

See also my experiments with GPT-3 on sane prompts, which have wildly varying quality even after generating them in bulk: https://github.com/minimaxir/gpt-3-experiments https://github.com/minimaxir/gpt-3-experiments Creative writing hasn't been one of the super-hyped use cases by OpenAI for the OpenAI API outside of AI Dungeon, surprisingly. For just random generation, the necessary curation can detract from the time-savings advantages. (as an aside, the API is also extremely expensive for long-form content to the point I'm not sure how the economics work for these startups even with charging monthly fees). I'm more bullish on small bespoke models for a given use case, which is what I spend my time researching. (example: a ~1M parameter GPT-2 Magic: the Gathering card generator: https://colab.research.google.com/drive/1VOt090UzvltoBgMdUZmU5vwhi4X-6E_a?usp=sharing https://colab.research.google.com/drive/1VOt090UzvltoBgMdUZm... )

YeGoblynQueenne 6y ago
Generating M:tG cards is interesting to me. Can we talk about this a bit? :) My degree dissertation was an M:tG expert system that included a hand-crafted parser and generator for (a small subset of) ability text [1]. My Masters' thesis was a grammar induction system trained on M:tG cards [2]. And I finally managed to sneak some M:tG grammar induction in my papers as a PhD student [3]. I had a quick look at the colab of your project. If I may be critical, the examples you have are ... so-so. I've seen previous attempts to generate M:tG text with neural nets, starting with the old project in M:tG Salvation that became Roborosewater and that used LSTMs if memory serves. Results have always been a mixed bag that looks promising but always falls down holes even with hand-curation. This still seems to be the case with your project. A couple of examples (you probably know already): −2: Put a +1/+1 counter on each other creature you control. "Each other creature" should refer to a card with the (super) type "creature" but this card is a "planeswaker". +1: Each opponent chooses a creature they control. If that creature was a copy of that creature, each other player reveals their hand, chooses one of those cards, then puts it into their hand, then shuffles. Which creature was that creature a copy of? −10: Target opponent sacrifices a creature. −9: Each opponent sacrifices a creature. Representing costs in a meaningful manner is a constant problem in every M:tG generator I've seen. The problems I highlight above are not with grammaticality, which is certainly a big step forward with respect to the past. But many of the abilities still don't make a lot of sense, or don't make sense to be on the same card, or have weird costs etc. If you want your project to be something genuinely new, I have to say I don't think it's there yet. My intuition is that it would take a lot more than language modelling to generate M:tG cards that make enough sense that it's more fun to generate them than create them yourself. I think it would be necessary to have background knowledge of the game, at least its rules, if not some concept of a metagame. It might also help to re-train standard NLP pipelines specifically for M:tG (e.g. some kind of ability-text specific anaphora resolution might help with "that creature" above). Also, I note that the new online version of the game is capable of parsing cads as scripts in a programming language using a hand-crafted grammar rather than a machine-learned model [4] [5]. So it seems to me that the state-of-the-art for M:tG language modelling is still a hand-crafted grammar. __________________ [1] https://github.com/stassa/Gleemin https://github.com/stassa/Gleemin - unfortunately, doesn't run anymore after multiple changes to Prolog interepreters used to create and then port the project over. [2] https://github.com/stassa/THELEMA https://github.com/stassa/THELEMA - should work with older versions of Swi-Prolog, unfortunately not documented in the README. [3] https://link.springer.com/article/10.1007/s10994-020-05945-w https://link.springer.com/article/10.1007/s10994-020-05945-w - see Section 3.3 "Experiment 3: M:tG fragment". Cringe-inducing to read just a few months after writing, sorry. [4] https://www.reddit.com/r/magicTCG/comments/74hw1z/magic_arenas_gre_the_groundwork_for_future/do0rvfe/ https://www.reddit.com/r/magicTCG/comments/74hw1z/magic_aren... [5] https://www.reddit.com/r/magicTCG/comments/9kxid9/mtgadispersal_bugged/e73pr9u/ https://www.reddit.com/r/magicTCG/comments/9kxid9/mtgadisper...

notum 6y ago

I'll admit, I just scrolled to the bottom to see if the article ends with "This piece has been written by GPT-3". It doesn't.

Animats 6y ago

GPT-3 is already better than most blog comments, many bloggers, and most online text porn. Maybe most ad copy. Many pundits. Some college professors. It has no clue what it's talking about, of course. It's terrible at fact-based material, because what comes out looks plausible, but contains made-up facts. Some facts will be correct, because they were mentioned a few times in the training database. Others are just plausible words strung together. What it really demonstrates is that much of literature has less behind it than was previously supposed.

FeepingCreature 6y ago
I expect GPT-3 to be amazing for people who want to write stories or articles but are really terrible at one specific aspect of the writing process.
- sbierwagen 6y ago
  https://slideplayer.com/slide/6081467/18/images/12/The+scale+of+intelligent+minds%3A+a+parochial+view..jpg https://slideplayer.com/slide/6081467/18/images/12/The+scale... It depends on if you think writing ability is more like the top graph, or the bottom graph. If it's like the top graph, humans have plenty of time to adopt AI tools, and incorporate them into the writing process. If it's like the bottom graph, then GPT-4 comes out in 12 months and is human equivalent, and GPT-5 comes out a year after that and is a better writer than any human could hope to be.
  - Animats 6y ago
    Progress so far is that GPT-1 could produce a coherent sentence but stopped making sense after a few lines. GPT-2 could produce a coherent paragraph, but several paragraphs strung together made no sense. GPT-3 is coherent for about a page or screen, but not a chapter. A page is good enough for the clickbait industry.
bpodgursky 6y ago
I have a friend who uses GPT-3 to generate startup marketing copy and customer-facing emails. He just prompts with the company description and "The email which is most effective at getting customers to sign up for <startup> is:" It reads as good or better than a human would do. Maybe not better than the BEST human marketer would do, but far better than average, and instant + free(ish).

mattikl 6y ago

Rewinding a bit from computers doing all the writing, are you using any tools that help you write considerably better (not just spelling/punctuation checking)? As a non-native English speaker I would like to have a tool that points out my nonstandard usages of language. I think it could be something as simple as parsing the text and searching it against a corpus, but I have no idea how many "unique structures" a text typically contains where no one else has used those same exact words. AI could help to suggest better ways to formulate the text.

camillomiller 6y ago
As a fellow non-native who accepted not to strive for full assimilation, my suggestion is to do the same. You can refine your writing by reading a lot. Books like “The elements of style” and “On writing well” can help with grasping those structural elements you might be struggling to identify. Apart from that, our brain has been shaped by our native language, and we can’t really delete that. Which is great news! It means we can express concepts in English with structures and flows that, if grammatically correct, have the advantage of sounding original, different, sometimes exotic. One tool I use is Grammarly. I think it’s mostly crap, but if used properly it helps you identify the more blatant mistakes. Oh, and by the way, if you need a non-native speaker ego booster, try and read native speakers’ first drafts of anything.
- mattikl 6y ago
  That's a good point of turning your native language into an asset. The goal of communication is to be well-understood. That's so easy to forget that and strive for something more. Good writing also keeps the readers engaged, maybe entertained, but is there much more to that?
- thaumasiotes 6y ago
  > It means we can express concepts in English with structures and flows that, if grammatically correct, have the advantage of sounding original, different, sometimes exotic. Yeah, that happens all the time. It always strikes me as unfair when someone comes out with an ostensibly perfect sentence and the result is "well, you didn't make any mistakes; all the words mean what you want them to mean and all the grammar is used correctly. But the sentence is still wrong; that's just not how we say that."

8bitsrule 6y ago

Couldn't help but notice that 'Xanadu' uses a lot of rhymes ... GPT apparently didn't notice that constraint. And then there's: "While all the stars that round her burn’d, Bow’d to the ground and based their fires. To the one ever-branching cloud" When 'base' is used as a past-tense verb, we expect a prepositional phrase ... before the next period. The preposition is never 'to'. Doesn't matter; the period stopped the English part. "That blew and drifted—blow and drift;" What? "If you told me that Coleridge wrote it, I would believe you." NExt.

gwern 6y ago
> Couldn't help but notice that 'Xanadu' uses a lot of rhymes ... GPT apparently didn't notice that constraint. Due to an obscure low-level technical decision (encoding text into BPEs), phonetics are erased from the GPT training data, and so therefore neither GPT-2 nor GPT-3 can learn to rhyme or make puns (they can only memorize relatively common pairs by bruteforce): https://www.gwern.net/GPT-3#bpes https://www.gwern.net/GPT-3#bpes I put a good deal of effort into testing this, including "Kubla Khan": https://www.gwern.net/GPT-3#samuel-taylor-coleridge https://www.gwern.net/GPT-3#samuel-taylor-coleridge Just doesn't work, and it never will until OA trains a model using some sort of better encoding which preserves phonetic information. (Character-based, unigram LM, or BPE dropout might all work, we don't know yet.)

caconym_ 6y ago

The lesson we should take from GPT-3 is that we spend way too much time reading text that isn't really saying anything, and/or failing to find meaning in what we read. It's a neat trick, and I think it could absolutely replace much or most of the writing on the internet today. But is it good writing? No, not really. At least, not in the sense of writing as producing anything more substantive than slick prose with a vague, meandering context running through it. I think the next step (say, being able to produce short stories with actual literary value) will be substantially more difficult, mostly because it's substantially more difficult for human writers too. It takes learned abstract knowledge and deliberate intention to write something that's coherent and engaging and longer than a few hundred words, and that skill is separate from the intuitive prose generation ability that's essentially unconscious for practiced writers and comes easily to many avid readers (i.e. trained on a large corpus) who take up writing. It will be interesting to see how producers of actual substantive content incorporate these tools into their processes, if at all. I guess I can imagine using it to generate descriptions, for instance, but how does it do with dialogue? Probably not very well.

DyslexicAtheist 6y ago
> But is it good writing? No, not really. agree. I'd say it's impossible to solve. Where would we even place our yardsticks: Has society been able to get any better (than we used to) at assessing and categorizing what "constitutes" great writing? If so in what relation could that be measured to our past performance especially that many no longer read today due to our loss of attention span? (most won't even read this comment because it's way too long). People are terrible at spotting some of the best writing when it happens. History is full of writers who's work could only be understood and appreciated after their death.[1] Death makes their work a finite commodity. There is no more chance asking the author "what they meant". Others get shunned because they were too radical, (or too ahead of their times). Great writers as in "a radical new style" or as "for their radical ideas" get only credited after decades (if not centuries) post mortem. When it's too upsetting (because it's true) we need the distance of history and later generations to appreciate it[2]. Even "what" we comprehend and then appreciate changes over time, and within the same person with every time we read it[3]. [1] 6 people centuries ahead of their time https://www.oxford-royale.com/articles/6-people-centuries-ahead-time/ https://www.oxford-royale.com/articles/6-people-centuries-ah... <- there are hundreds of wrtiers like this (it is just the top result google gave me). How would an AI spot it when all AI was created "in our image". And if it did spot it how would the AI be able to convince us that this is worth a closer look instead that we need to destroy the AI? [2] The Last Messiah by Peter Wessel Zapffe https://philosophynow.org/issues/45/The_Last_Messiah https://philosophynow.org/issues/45/The_Last_Messiah <- say something mediocre to ensure you get published and ranked in "best new books". But saying something profound can potentially get you ignored (at best) and killed (at worst). This ties back to[1] [3] James Joyce Dubliners https://archive.org/details/dubliners00joycrich https://archive.org/details/dubliners00joycrich / https://en.wikipedia.org/wiki/Dubliners https://en.wikipedia.org/wiki/Dubliners <- there are hundreds of examples like Dubliners but they are all different books depending on who reads them and when. the retrospect and integrating what we read (and then re-read over and over at various stages of our lives) into our own reality is what is makes it great. yet even if some minority of people does practice this, it's still highly subjective. how would an AI provide us with value (e.g. if it can the value it creates would only benefit itself but never others) __ I'm not saying AI is incapable of better writing than AI is now, but it would only remain mediocre and never outperform humans. Also the writing can't stand or be judged by itself. It can only be valued and appreciated through human mind. Art (and many books are art) has the same problematic relationship with AI. John Berger's "Ways of seeing" talks about this (but so does a lot of philosophy) https://www.youtube.com/watch?v=0pDE4VX_9Kk https://www.youtube.com/watch?v=0pDE4VX_9Kk

cjohansson 6y ago

The AI generated paragraph made no sense to me and did not sound like the first. AI is like a narccisist, every other day people say it's useless and then the next it's the savior of the world.

ayushchat 6y ago

I've actually used tools like copy.ai and converstion.ai for a couple of copywriting projects and found them to be very useful. Of course the output isn't perfect, human intervention is required, but it is way better than doing the writing myself from scratch. I think going forward we are going to see a lot of human + AI writing projects instead of purely human or purely AI.

nxpnsv 6y ago
* you mean https://www.conversation.ai/ https://www.conversation.ai/ ?
- dave84 6y ago
  I'd imagine it's https://www.conversion.ai/ https://www.conversion.ai/
  - nxpnsv 6y ago
    Yes, looks better... thank you!
- ayushchat 6y ago
  yes right, typo I meant https://www.conversion.ai/ https://www.conversion.ai/ Their AI assistant is called Jarvis

db39 6y ago

This is something I've been deep into recently. Just a couple of days ago I published a short paperback story (grimtalesbook.com) written and illustrated by AI - GPT-3 for the writing in this case. I liked the analogy aimor made in this thread between the level readiness of autonomous driving and autonomous writing. From my experience that feels right, we're not at L5 yet - you can't just let the AI write continuously because you'll just end up with junk wrapped around some nuggets of gold. In the future if we get to 'L5' writing, your child will have the ability to say 'OK Google, tell me a story about my adventures as a dragon slayer' and Google will respond happily with a perfect bedtime story. And the next day, you'll get a physical copy of that story in the mail. Luckily, that future isn't here yet. The way Grim Tales was written with GPT-3 was by using some human intervention to keep the AI on track to write a cohesive narrative with a plot and dialogue. A couple of rules like removing anything that doesn't progress the narrative, and keeping the narrator in first person meant that the story was far more readable and didn't meander around like common examples. Right now - for professional writers - AI is probably good enough as a co-pilot used to create alternate scenes at points in the narrative, or as a tool to help with writers' block.

WesolyKubeczek 6y ago

I have a suspicion that, since GPT-3 gorged its enormously swollen fuzzy digital belly on a lot of real prose, it might, when prompted, occasionally throw in a piece of the prose in question, or a whole original passage, because to its fitting function, it would be a suitable thing to do. But then again, aren’t we all doing the same, regurgitating bits and pieces that we heard or read before and liked? Don’t we all sometimes, when busy with our own thoughts and worries, occasionally spout out oddly appropriate responses to what our spouses are speaking, without actually listening to and processing the conversation, maybe because we heard it all so many times we can readily predict what they are going to say? And then we agree that we need to go take the alligator from the repair shop. However, if anything, GPT-3 is mostly showing how little meaning there is, but in how many words. Never mind me, I’ll pour myself another one.

yesenadam 6y ago
> Don’t we all sometimes, when busy with our own thoughts and worries, occasionally spout out oddly appropriate responses to what our spouses are speaking, without actually listening to and processing the conversation, maybe because we heard it all so many times we can readily predict what they are going to say? Uh.. no, not me. That sounds very strange to me, and sad. I always listen to everything she says. And expect the same back. Otherwise it wouldn't be a good relationship, I think. (Maybe I'm super-naïve!..or something.)
claudiawerner 6y ago
>Don’t we all sometimes, when busy with our own thoughts and worries, occasionally spout out oddly appropriate responses to what our spouses are speaking, without actually listening to and processing the conversation, maybe because we heard it all so many times we can readily predict what they are going to say? No. Sometimes I can predict what a friend will say but I won't actually say it. The more I think about it, the more I realise that AI at the moment is the unlistening (but hearing) spouse. And just like it seems we can engage with such a person (though can only rarely), so too can the AI trick us ('simulate') that behaviour.
YeGoblynQueenne 6y ago
>> But then again, aren’t we all doing the same, regurgitating bits and pieces that we heard or read before and liked? We can do that. We can also not do that. Language models can only do that. My comment is stripped of nuance (language models don't do that exactly) but it should point at the difference: humans can be as dumb as bricks, or as smart as humans. Language models don't even come close to our brighter moments. They don't even come close to the brighter moments of our pets, and our pets can't generate human language. You have kids? Think of whom you'd like to spend the next decade or two around: a five year old human or GPT-3? Which one would you find more interesting, original, curious, do you think? I think people who are the most excited about GPT-3 are the ones who are used to even dumber, more boring interactions with computers than the repetitive and unoriginal text generation of language models. Or perhaps it's humans who are used to boring interactions with other humans. Technology has done some weird things to us of late I fear.

bambax 6y ago

Many good and prolific writers of the past used ghost writers who prepared drafts and idea lists for them, in effect making their work more like that of editors (it is said Alexandre Dumas worked like this -- it's one of the only reason to explain his spectacular productivity). So if AI is capable of doing some prep work, why not? When writing fiction, my struggle is usually not in the writing itself (making sentences) but in figuring out what to write about (the content of a scene). An AI that would generate scene ideas from an input of, say, a list of characters and plot points, would be very helpful. Outputting sentences and paragraphs, not so much (I think). Also, the article seems to conflate Sudowrite and GPT-3 but it would be interesting to understand the difference between the two; is Sudowrite just a wrapper over the GPT-3 API that forwards requests directly? If not, what is its added value vs hitting the API directly?

jamesjyu 6y ago
Founder of Sudowrite here. We are built on GPT-3 but we have a bunch of things we do on top that optimizes Sudowrite for the narrative fiction/non-fiction use case. For example, we have a description tool that, given a highlighted phrase, gives you options for telling phrases in 5 senses and metaphorical uses. Another tool we have analyzes a story premise and gives you possible twist and story turns, based on the genre of your story, and also gives you possible characters that could inhabit that world, complete with both external and internal character descriptions. I agree that for writing stories, the hard part is in the forest and not the trees (although this isn't true for everyone). And systems like GPT-3 can help with that, too, simply because it's level understanding of text is approaching that of a human.
- bambax 6y ago
  Interesting! Thank you for answering. How long is the waiting list to join Sudowrite beta?
  - jamesjyu 6y ago
    It’s gotten quite large, but we plan on sending more invites out soon. We are prioritizing folks who are working writers and have an immediate project they’re working on that could benefit from collaborating with an AI. If this sounds like you, sign up on the beta form and also send me a note about how you’ll use Sudowrite to jamesjacobyu@gmail.com

ta_ca 6y ago

until a few minutes ago i was a skeptic. now i think about it... this gpt business is not only AI but it might be a way to AGI. why? it mimics us perfectly that each of us can fill books about the things we have zero understanding of.

jgalt212 6y ago

> In July of 2019, Microsoft invested a billion dollars, which allowed OpenAI to create a supercomputer with two hundred and eighty-five thousand C.P.U. cores, ten thousand G.P.U.s, and four hundred gigabits per second of network connectivity per server. > Gupta imagines the product turning into a resource that writers will pay fifteen to twenty dollars per month to use. That seems like a great deal to sublet some portion of all those resources.

troelsSteegin 6y ago

Are people getting better at reading? A deep read is an immersive, flow experience. But to engage in reading you have feel you have the time and focus to commit to reading and then not be interrupted. I don't want for machine generated narrative, I want more time spent deep reading. I do way more scanning and skimming now than deep reading - authoritative source, credible, what's new, what's useful - it's all critical thinking and done as efficiently as I can make it. It feels like work. I always wonder if more structuring over the stuff that I read - entity and concept extraction, thesis extraction, separating facts from opinions, a topic-centered feed - would be better. I haven't found or built the right experience yet. To me, narrative generating models are interesting only as a step toward a better critical reading experience, and the win there would be generating useful abstractions that help one understand faster. Arguably, writing style is one such abstraction. Maybe there's potential in being able to generate readable narratives from things that people have a hard time describing, like datasets. Then the grail is business dashboards composed as sonnets.

jamesjyu 6y ago

Oh wow, just seeing this post made it to the front page. Founder of Sudowrite here, happy to answer questions.

libertine 6y ago
I don't mean to put on you the spot, but I have been wondering about this: Do you think that such tools will have the eyes of the regulators on top of them due to copyright issues, since the data used could have a lot of copyrighted material? I haven't yet made my mind about this: is the output original content, or just shuffled proprietary content?
- jamesjyu 6y ago
  (Caveat: IANAL) As far as I know the issue of copyright and AI generated content hasn’t been tested in the courts yet. There are two issues: 1) The author inadvertently uses output from GPT-3 that happens to be close enough to copyrighted material to be a problem. This is why we recommend to authors do two things: a) enter as much of their own original work to prompt Sudowrite and b) edit and incorporate the AI output organically into their work. The chance that GPT-3 outputs verbatim copyright work in that case is small. As we grow, we also plan to incorporate plagiarism checks in the background. 2) Someone could challenge the author’s copyright on a piece of work that was primarily written by GPT-3. This has not been tested in the courts, and I suspect it will come to head in the next few years. For me, the question of who owns copyright is dependent on how much work the author has done to the AI output (there are some parallels to the Monkey Selfie case [2]). For example, Stephen Marche (the author of the new yorker post from OP) also wrote a short story [1] using Sudowrite, in which he claims that 17.1% of the final words were from the AI. But of course, he has done non-trivial amounts of work to edit and incorporate those words and made it his own. So in this case, he def deserves ownership. All of our Sudowrite authors use the AI similarly in their process. However, what if an author uses vanilla GPT-3 with no humans int the loop to generate hundreds of biographies of famous people based on wikipedia entries? I don’t have a good answer to this. It shouldmay come down to how much the author has done to guide GPT-3 and/or pre-post-process it. Here’s further reading: http://rutgerslawreview.com/wp-content/uploads/2017/07/Robert-Denicola-Ex-Machina-69-Rutgers-UL-Rev-251-2016.pdf http://rutgerslawreview.com/wp-content/uploads/2017/07/Rober... https://core.ac.uk/download/pdf/77599763.pdf https://core.ac.uk/download/pdf/77599763.pdf [1] https://lareviewofbooks.org/article/the-thing-on-the-phone/ https://lareviewofbooks.org/article/the-thing-on-the-phone/ [2] https://www.npr.org/sections/thetwo-way/2017/09/12/550417823/-animal-rights-advocates-photographer-compromise-over-ownership-of-monkey-selfie https://www.npr.org/sections/thetwo-way/2017/09/12/550417823...

bulldog13 6y ago

What is current state of the art that a user can actually access ? For example, GPT-3 requires open ai beta access. Is it still GPT-2 ?

SpeakerFrThDead 6y ago

Does anybody know what the author means when he asserts that "nearly all writers he knows don't like writing"? Seems contradictory to everything I've heard about the iteration loop between practice, play, and discovery that characterizes mastery in any craft, be it programming, writing, music, sports, and so on.

konfuzio 6y ago

GPT Neo works well for drafting emails. Find our approach here: https://www.helm-nagel.com/open-sourced-gpt-Neo-writes-individualized-emails https://www.helm-nagel.com/open-sourced-gpt-Neo-writes-indiv...