24 ms·

OpenAI's GPT-3 may be the biggest thing since Bitcoin

• rvz 6y ago
  > I further predict that this will spark a creative gold rush among talented amateurs to train similar models and adapt them to a variety of purposes, including: mock news, “researched journalism”, advertising, politics, and propaganda. The first mention of 'Elon Musk' (who left the board) and this sentence alone gave me the tip-off that GPT-3 had generated that (and the whole blog) and it's following prediction makes no sense. Sure, it may be used for nefarious purposes, but no-one can train GPT-3 in any acceptable time except for those with access to large GPU/ASIC compute power (OpenAI, Microsoft, Google, NVIDIA, etc.) Without the model, it is not possible to adapt it to any other purpose, unless OpenAI does it for them. Without a detection mechanism, it is very dangerous. Nice try and a great GPT-3 hype experiment, (mostly by friends of OpenAI). I look forward to the day that GPT-3 gets proper scrutiny from the actual wider tech industry before we can safely use it with detection methods.
  • monocasa 6y ago
    Propaganda, advertising, and politics (arguably all the same thing) absolutely have access to the funds for this. Facebook and Google, the biggest investors in state of the art ML are advertising companies. Their customers have the funds to do this independently.
    • confeit 6y ago
      Giving certain people, highly visible on social media, pre-public access to the model, and letting them cherry pick their completions to post without the prompt or amount of tries, is a smart form of propaganda/hype building/PR management, that we have come to expect from "GPT-2 is too dangerous to release" openAI Sometimes I forget that, while this model was created by scientists, and released with a scientific paper, it is essentially a for-profit business product, and such cheap tricks deserve harsh criticism.
      • derefr 6y ago
        > Sometimes I forget that, while this model was created by scientists, and released with a scientific paper, it is essentially a for-profit business product, and such cheap tricks deserve harsh criticism. Sure, but this is akin to seeing bad science journalism and tarring the science itself with the same brush. GPT-3 still factually has certain properties, independently of anyone making grandiose assertions about those properties. What those properties are, we can only say slightly—e.g. we know it’s capable of generating certain texts eventually, among an unbounded corpus of other texts it may have generated that were then human-discarded. But the fact that it can generate those texts at all—faster than brute-force, I mean—is an interesting fact on its own, worthy of scrutiny independent of whatever airier claims are being made.
        • BickNowstrom 6y ago
          It is certainly impressive, and I don't want to discard GPT-3. Just critiquing the (smart) release: make a select few feel special by giving them API access, and watch your product dominate the tech - and news cycle for weeks. You'll have VC money in the bank before showing actual worth or business value. Maybe a bit simplistic, but I view GPT as a Markov chain text generator, operating on word vectors instead of word tokens, and having a larger look-back. It's like a child copying a joke, because she heard adults laughing about it, but she does not understand the punchline. You wouldn't say that child understands or even displays humor, despite substituting "horse" with "donkey" when retelling the joke.
          • Filligree 6y ago
            If you want to play with GPT-3, you can do so right now. Go to https://play.aidungeon.com https://play.aidungeon.com Make an account, and select the "Dragon" model. That's GPT-3. I've spent ten hours playing with it over the last two days. It isn't perfect, and it feels short of the hype it's generating about itself, but it's an amazing leap nonetheless. It really seems to have an understanding of causality, biology, all sorts of fictional themes... It isn't perfect. You frequently have to back it up and try again. Unless you make good use of the site's long-term memory function, it'll forget anything that happened over a page ago, and a lot of the time its idea of what should happen next doesn't match the plot I had in mind. I'm getting better at that. However, as a writer myself, I can say that this is just as true for human writers as well. For every final draft you see there are ten discarded ones, and a hundred that never made it to paper. Viewed that way, GPT-3 is actually much better at the core part of writing than I am! It's more creative, it uses English better, it's better at matching the narration to the characters than I am... It's just that this isn't enough. It's missing a full model of the world, and it doesn't know how to look at what it's written and decide if it matches its intent, or whether it'll break consistency or get in the way later. It doesn't have an intent. It doesn't know about consistency. But that's also true for that part of me. GPT-3 isn't a human-level writer. What I've determined, however, is that it's a huge part of one, and it's more than good enough to fulfill the role of that part already. Now we just need the other nine tenths.
  • graham_paul 6y ago
    > no-one can train GPT-3 in any acceptable time except for those with access to large GPU/ASIC compute power (OpenAI, Microsoft, Google, NVIDIA, etc.) Any state actor has access to large compute power
    • briga 6y ago
      But it might be hard to attract talented AI engineers to live in, say, North Korea
      • acdha 6y ago
        What about North Koreans desperate to be part of the privileged elite? They can take courses and learn just like anyone else.
        • nefitty 6y ago
          I’m genuinely curious how accessible that sort of material is to the average North Korean.
      • Rebelgecko 6y ago
        How much of a choice do they have? I can't imagine many of the top engineers in North Korea are allowed to leave
        • ekianjo 6y ago
          When most of your population is borderline starving the number of top engineers can be counted on very few hands in NK.
          • saagarjha 6y ago
            That being said, North Korea has a fairly decent cyberespionage program from what I hear.
          • graham_paul 6y ago
  • smallnamespace 6y ago
    > no-one can train GPT-3 in any acceptable time except Gwern, who has spent probably as much time with GPT-3 and GPT-2 as any 'amateur' out there, is publicly out there saying that for most use cases, GPT-3 + creative use of prompts gets you better results than GPT-2 with finetuning. That's an amazing capability that Gwern elaborates on more here: > A new programming paradigm? The GPT-3 neural network is so large a model in terms of power and dataset that it exhibits qualitatively different behavior: you do not apply it to a fixed set of tasks which were in the training dataset, requiring retraining on additional data if one wants to handle a new task (as one would have to retrain GPT-2); instead, you interact with it, expressing any task in terms of natural language descriptions, requests, and examples, tweaking the prompt until it “understands” & it meta-learns the new task based on the high-level abstractions it learned from the pretraining. This is a rather different way of using a DL model, and it’s better to think of it as a new kind of programming, where the prompt is now a “program” which programs GPT-3 to do new things. https://www.gwern.net/GPT-3#prompts-as-programming https://www.gwern.net/GPT-3#prompts-as-programming Almost all the GPT-3 results you see on Twitter are via the OpenAI API - no finetuning, only prompting. That implies... > Without the model, it is not possible to adapt it to any other purpose, unless OpenAI does it for them ... that we're actually very far from plumbing the possible ranges of behavior of GPT-3 with different sorts of prompting. This is a new ballgame folks. The old rules don't quite apply here.
    • yomly 6y ago
      I don't really understand your post (and don't know much about GPT3) are you suggesting that the model is stateful in that it can continue learning from successive prompts? Maybe you can elaborate on this: >instead, you interact with it, expressing any task in terms of natural language descriptions, requests, and examples, tweaking the prompt until it “understands” & it meta-learns the new task based on the high-level abstractions it learned from the pretraining. Or are you suggesting that it has such a deep network of abstractions that once a user starts to map that out, the mileage they can extract back out of the model via prompts is very exciting.
      • walleeee 6y ago
  • threeseed 6y ago
    > no-one can train GPT-3 in any acceptable time except for those with access to large GPU/ASIC compute power That's not true. a) Premium AWS customers, for example, can request to have instance limits removed which then gives you access to all of the AWS GPU-enabled instances available worldwide. b) People ramble on about GPUs but Intel DLBoost/AVX-512 enabled CPUs can get you comparable performance to a medium end GPU in many situations. That then opens the door to training across all of the cloud and VPS providers. Money is the limiting factor here not available compute resources.
    • thu2111 6y ago
      Do you have any links I can read in the latter claim? Using avx512 for ai with comparable power to a GPU?
  • sinity 6y ago
    Uh-huh. Except anyone can verify the results themselves, sans specific ones like code generation - through (paid version) of AI Dungeon. Amount of people confidently posting bullshit on Hacker News is astounding. Reacting like everything we know about GPT is just a bunch of tech demos. Supposedly everyone who has access to the API is just a shill. Eh.
    • Tarrosion 6y ago
      I just spent an hour or so playing around with the paid version of AI Dungeon, and was super unimpressed. It's pretty fun for a moment, sure, and I assume some really heroic work went into building it. I'm not saying the creators did a poor job so much as the task is really hard and the final result is...lacking. The "Dragon" (GPT-3) engine responds reasonably to any particular input, but clearly lacks a coherent state of the world. Objects appear and disappear; plot cues are given and then can never be summoned again if not immediately grabbed, environments change dramatically without explanation, etc. Do you feel otherwise?
      • ilaksh 6y ago
        Right. It has some type of language ability, but no world modeling. So overall it really doesn't make sense. But since that is so obvious, I assume many people are trying to figure out how to improve it. So I am excited to see if they can make progress in the next few years. It is going to be quite difficult though. I think it might require integrating a totally different type of subsystem, if it is possible at all. But the ability to make realistic sounding language is a step forward it seems to me.
  • killerstorm 6y ago
    Why do you believe they won't release the model? It's Open AI after all. Google, NVIDIA and Microsoft also make their models freely available. Google already trained a model which is bigger than GPT-3. There might be some delay, but no fundamental problem with it. Fine-tuning the entire GPT-3 is impossible on a single GPU. But it's still possible to fine-tune specific layers. Plus, I'm sure somebody will release a distilled version of it which is more manageable.
    • polyanos 6y ago
      >Why do you believe they won't release the model? It's Open AI after all. Because they want to commercialize the model "to cover the costs of research", it's stated in their blog post FAQ[0]. >Google already trained a model which is bigger than GPT-3. Source?, a fast Google search yielded no results and I'm curious. [0]https://openai.com/blog/openai-api/ https://openai.com/blog/openai-api/
      • killerstorm 6y ago
        They commercialize the API. The model is so large you can't run it on a single GPU, AFAIK. OpenAI developed an infrastructure to run the model efficiently. So many companies would rather use API than deploy the model on their own hardware. They will probably release the baseline model, not model optimized for deployment. There are many optimizations possible such as precision reduction, pruning, distillation, and they don't have to share these optimizations. > a fast Google search yielded no results and I'm curious. List of pre-trained models on huggingface: https://huggingface.co/models https://huggingface.co/models You can see some of them are prefixed with "google". Also ALBERT is from Google Research. That's just natural language processing, I dunno what they have in other fields.
      • nschucher 6y ago
        They trained a >600B parameter translation model (GPT-3 is 175B parameters). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (https://arxiv.org/abs/2006.16668 https://arxiv.org/abs/2006.16668)
• minimaxir 6y ago
  I published a response today to the sudden hype urging people to temper their expectations for GPT-3 a bit: https://minimaxir.com/2020/07/gpt3-expectations/ https://minimaxir.com/2020/07/gpt3-expectations/ GPT-3 is objectively a step forward in the field of AI text-generation, but the current hype on VC Twitter misrepresents the model's current capabilities. GPT-3 isn't magic.
  • nbardy 6y ago
    One of the biggest issues is with cherry-picking. Generative ML results benefit greatly from humans sampling the best results. They are capable of producing astonishing results but don’t do this consistently this has a huge impact on any effort to productize. For example I’ve seen quite a few examples of text->design, text->code, with GPT-3 you could build a demo in a day, but the product will probably be useless if it’s not delivering results 50%+ of the time
    • drcode 6y ago
      I agree, but on the other hand one has to be careful not to be blind to the obvious power of a new technology, simply because it cannot be immediately turned into $$$.
    • quotemstr 6y ago
      I think too few people are taking into account how inscrutable and inconsistent human creative-output results can be. We critique GPT-3 on the basis of it sometimes producing bad results --- but don't we all? Take poetry, for example. "The Complete Works of X", for any X, will probably contain a majority of forgettable or just bad works. What we remember from any author X is his cherry-picked best output. Likewise for ML systems.
      • yyyk 6y ago
        The hype/scare re GPT-2/3 (etc.) is not for their poetry output, but rather for its potential for mass propaganda, telemarketing and so on. We can already get humans to do this stuff, all GPT could give is scale (that's no small deal). However, if the output needs to be curated and edited by humans, the scale and automation is gone - we just get a different manual process, with a modest improvement to speed at cost of some decline in quality, and that's not very impactful.
• evmar 6y ago
  For those responding to the title or first half of the article, you are missing the point! Edit: the comments that are reacting to the article's content are from people who didn't read it to the end, wow.
  • bearer_token 6y ago
    It's truly incredible. Demonstrable proof of passing the turing test. Or - maybe the commentators are bots too. It's now impossible to tell.
  • anonred 6y ago
    I do sometimes wish that HN provided an optional “proof of work” field for comments to differentiate between those responding to the title and those who actually took the time to read the article. Something like a “verified” check mark or toggle button. I’ve found that discussions are most insightful when all participants have “done their homework” so to speak. So while it’s amusing that half of the folks didn’t read the relatively short post that was linked here plus the twist at the end, it’s a sad state of affairs when many (most?) online discussions have devolved to this.
    • nullc 6y ago
      It would be nice to simply eliminate the advantage that uninformed posts enjoy. If you shoot from the hip, your comment is first and will be seen by a lot more people. A forum could instead queue up comments and only display them after a half hour has passed. People who come to the article later will still be at a disadvantage but at least this would be something. On the other hand, you'd probably get a lot of duplicate remarks... which would be less than interesting.
• drcode 6y ago
  Personally, in 2014 I had put the odds of seeing human-level AI during my lifetime at around 10%. Given the latest developments, I would put the odds at fifty-fifty within the next decade.
  • graham_paul 6y ago
    > Given the latest developments, I would put the odds at fifty-fifty within the next decade. thing is technological development is not linear, you can't predict future development based on the last n decades. You can't assume an AI winter is not coming because it most likely is
    • staticassertion 6y ago
      > because it most likely is Is this not you making a prediction, the very sentence after saying one can not predict the future?
      • perl4ever 6y ago
        Saying an AI winter is coming isn't saying how long it will be for or what the magnitude will be.
        • staticassertion 6y ago
          OK, so what is it saying?
          • perl4ever 6y ago
            That progress won't be steady and predictable.
      • graham_paul 6y ago
        No. This is like saying that if today is not raining, a rainy day is coming
    • 6y ago
• Keyframe 6y ago
  disruptive potential comparable to that of blockchain technology. When I read this, I don't know, genuinely, if it's a _potential_ blockchain has/had or a realized potential, which I find hard to believe since it didn't disrupt anything apart gamblers so far.
  • sidcool 6y ago
    Reading the entire article is highly insisted.
    • Keyframe 6y ago
      I agree and I recommend the same.
  • postingawayonhn 6y ago
    The bot wrote the article and presumably it learnt that blockchain is disruptive from BitcoinTalk.
    • zerocrates 6y ago
      No, the human "author" supplied the "disruptive potential of blockchain" part.
• davidgerard 6y ago
  I dunno. There's a bot on /r/buttcoin that posts dumb bitcoiner quotes, and bitcoiners would keep showing up and arguing with it. So I think the Turing test bar's a little low in this domain.
  • drcode 6y ago
    /r/buttcoin is still around? You folks haven't conceded yet? :-)
  • Natsu 6y ago
    How do you know that they're bitcoiners and not someone training their own bot? :)
  • nullc 6y ago
    > I dunno. There's a bot on /r/buttcoin that posts dumb bitcoiner quotes, and bitcoiners would keep showing up and arguing with it. So I think the Turing test bar's a little low in this domain. If you're saying that because you understood that the article was claiming that GPT3 was impressive because it fooled people on bitcointalk.org, then I think you just demonstrated that GPT3 passed the Turing test with respect to cryptocurrency-critics. :) [The posting to bitcointalk.org was a fiction written by GPT3.]
• qeternity 6y ago
  Oh man. I don’t know where to begin. I’ll just say that this analysis is predicated on someone who thinks Bitcoin has proven anything in the real world. We’re way past BTC being a subculture experiment. Everyone knows about it. And still nobody (in the statistical sense) uses it.
  • deleted 6y ago
    [deleted]
  • gk1 6y ago
    You can begin by reading the article to the end. :)
  • cli 6y ago
    Are you referring only to the title?
  • vasili111 6y ago
    The main idea is at the end of article :)
  • mpfundstein 6y ago
    a lot of people use it a lot.
    • qeternity 6y ago
      No, not in the statistical sense.
  • golergka 6y ago
    Most people's who buy drugs use BTC, all street dealers completely disappeared, at least around here. That's not a small market.
  • fnord77 6y ago
    Queen Elizabeth I was president of the United States in 1600.
• deleted 6y ago
  [deleted]
• meredydd 6y ago
  I am deeply enjoying this comment thread - it's a bit of a Barium Meal [0] for determining how many people read (a) the headline, (b) the first paragraph, or (c) the whole thing before jumping straight into the compose box. Having read to the bottom, the quality of text generation there absolutely blew me away. GPT-2 texts have a somewhat disconnected quality - "it only makes sense if you're not really paying attention" - that this article lacks entirely. Adjacent sentences and even paragraphs are plausible neighbours. Even on re-reading more closely, it doesn't feel like the world's best writing, but I don't notice major loss of coherence until the last couple of paragraphs. I am now really curious about the other 9 attempts that were thrown away. Are they always this good?! [0] https://en.wikipedia.org/wiki/Canary_trap#Barium_meal_test https://en.wikipedia.org/wiki/Canary_trap#Barium_meal_test
  • jcahill 6y ago
    GPT-3 is a neat party trick. But the things that'll be done with web archives* in the next 20y will make it look like the PDP-8. ~love, a web archivist * GPT-3 is trained on one
    • deleted 6y ago
      [deleted]
    • h0p3 6y ago
      No pressure: feel free to ignore me, please. Would you mind elaborating? I'm interested in what you have to say (and, of course, feel free to say it privately if you prefer). I would like to even hear your dreams, wild speculations, or gut feelings about the matter.
      • jcahill 6y ago
        Sure, what do you want to know? I currently work on synbio × web archival. Some of us are cooking up futuretech aimed at storing all of IA (archive.org) in a shoebox. Others are working on putting archival tools in more normal web users' hands, and making those tools do things that people tend to value more in the short-term, like help them understand what they're researching, rather than merely stash pages. My ambitions for web archives are outsized compared to other archivists, but I'm fine with that. I'm looking beyond web archives as we currently understand them toward web archives as something else that doesn't quite exist yet: everyday artefacts, colocated and integrated with other web technology to an extent that they serve in essential sensemaking, workflow, and maybe security roles. Right now, some obvious, pressing priorities are (a) preserving vastly more content and (b) doing more with the archives themselves. A: The overwhelming majority of born-digital content is lost within a far narrower time-slice than would admit preservation at current rates, and data growth is accelerating beyond the reach of conventional storage media. So, for me, the world's current largest x is never the true object of my desire. I'm after a way to hold the world that is and the world to come. Ideally, that world to come is one where lifelong data stewardship of everything from your own genome to your digital footprint is ubiquitously available and loss of information has been largely rendered optional. This, of course, requires magic storage density that simply defies fundamental limitations of conventional storage media. I'm strongly confident that we're getting early glimpses of the first real Magic contenders. All lie outside, or on the far periphery of, the evolutionary tree that got us the storage media we have today. For instance, I'm running an art exhibition that involves encoding all the works on DNA. B: Distributed archival that comes almost as naturally as browsing is well within reach, and with that comes some very new potential for distributed computation on archives. One hand washes the other. One important thing to realize here is that, in many cases, you can name a very small handful of individuals as the reason why current archival resources exist. GPT-3 is cracking the surface by training on data produced by one guy named Sebastian, for instance. …i'm sorta tired and have to respond to something about every twitter snapshot since June being broken, though, so I'll pick this back up later.
• jbhouse 6y ago
  I thought the last paragraph was a bit odd, but chalked that up to my own ignorance of AI. Honestly, I am incredibly impressed by this... and a bit fearful
• k__ 6y ago
  I read, it's now all about "priming" and not "training" anymore. What does this mean? Where can I learn about priming?
  • drcode 6y ago
    With GPT-2, you had to (for instance) train the AI with thousands of poems for hours to get the AI ready to write its own poem. With GPT-3, you can just say "Here are three poems by Dr. Seuss about Grumpy Cat" and then it'll (sometimes) write some convincing poems.
• api 6y ago
  So it will be a massive bubble that sets billions of dollars on fire with little to nothing to show at the end?
• freeone3000 6y ago
  Still has some repetition and simplicity issues - article reads like it's written for an "eighth grade" reading level, and tends to repeat major points multiple times per line. There's also a lot less pronoun use than I'd expect in free-written text. These signs might prove useful in the months to come.
• 3wolf 6y ago
  It sounds like you have an output that has been generated by a recurrent neural network trained on a large corpus. The more I read about neural networks the more clear it becomes that there are some classes of them that just write some good stuff. The work of Andrej Karpathy on RNNs for poetry and other things comes to mind. Is your GPT-3 actually even a neural network, though? More research is needed! ^written in response to the blog by GPT-3
  • m3kw9 6y ago
    Weak
    • garbagetime 6y ago
      Beside the point, really - The interesting fact is that it's a lot better than anything we have previously had.
  • O_H_E 6y ago
    Worthy to note that as another comment mentioned, GPT3 was likely trained on pre-GPT2 data.
• Imnimo 6y ago
  I'm surprised that GPT-3 understands how to write coherently about GPT-3. Common Crawl, the bulk of GPT-3's training data, was collected even before GPT-2 was published. Maybe the Wikipedia dump they used was recent enough to contain a few references to GPT-2, but that's it. So if GPT-3 never saw any discussion of GPT-3, and probably almost no discussion of GPT-2, in its training data, how does it know what they are? The prompt text does give a hint that it's a language prediction model, but that hardly seems like it should be enough.
  • nullc 6y ago
    I mean, you could write coherently about GPT3 without ever hearing about it before-- right? You could absolutely write coherently about it with a bit of priming text, ... I mean, that is exactly what you're doing right now. :) Unless you work for open-ai, you've never actually seen GPT3... for all you know for sure it could be some amazing mechanical turk thing postprocessing GPT2 output :)
    • Imnimo 6y ago
      I suppose I could have guessed that GPT-2 is a plausible name for the predecessor of GPT-3, and the other predicted predecessor name (PTB) is not really correct. But I'm still surprised that it can hit a believable tone about its own capabilities. I don't think I could have done that if you had asked me before GPT-2 was published (which is essentially what we're doing if GPT-3's training data predates GPT-2). I think I would have under-estimated and it would be immediately clear to a reader familiar with GPT-3 that I didn't actually know what it could or could not do.
      • nullc 6y ago
        > I think I would have under-estimated Keep in mind, the prompt specified that it was the biggest thing since bitcoin and had disruptive potential. Given that you wouldn't write that it sucks. It also did underestimate its own performance: > ... nor do its predicted replies tend to be relevant or even grammatically correct. A prototype that had predicted replies that were convincing in most cases would be much more impressive than the GPT-3 I describe here, although that would probably require many years of training and many iterations of improvements on the model It essentially adopts a common convention for hype articles: A pile of breathless exaggerated hype and then a brief conclusion that suggests that the approach hardly works at the current time, at least if you're reading between the lines. :) I think it's more surprising that it didn't stray further into fantasy. I'd be interested in seeing the author's rejected samples.
• andreyk 6y ago
  Quite good, with some exceptional paragraphs like this making it at least read as bad writing - "I chose bitcointalk.org as the target environment for my experiments for a variety of reasons: It is a popular forum with many types of posts and posters." Still, it's very lucid and coherent until the last generated paragraph, not bad...
• bouncycastle 6y ago
  The irony is bitcoin didn't disrupt anything. Most of it is propped up by memes & hype, that made a few people rich. While it was impressive at the beginning and made many news stories, the technology itself is unsustainable, unscalable, with big usability problems. Disappointing to say the least. (Perhaps that's what the author of the article was subtly implying?)
  • vasili111 6y ago
    The article is not about bitcoin. Read it until the end :)
    • bouncycastle 6y ago
      I see, thanks :-)
• travbrack 6y ago
  The text generation quality is amazing, but the thing that's blowing my mind is how it can do math problems not seen in the training data. I've also seen a couple demonstrations of people generating simple apps with just a description.
  • polyanos 6y ago
    >I've also seen a couple demonstrations of people generating simple apps with just a description. And yet none of those people have released a demo, even though some said they would, multiple times. I'm still quite sceptical at those demonstration until I get to try it myself.
• simonsarris 6y ago
  I could tell this was GPT-3 because it reads like SEO-style sentence structure :/ > I imagine that similar results can be obtained by republishing GPT-3’s outputs to other message boards, blogs, and social media. I actually wrote a bit about this scenario and how it could explode comment sections to stonewall topics, which I'm calling Commentdämmerung: https://simonsarris.com/commentdammerung https://simonsarris.com/commentdammerung Still, I think a lot of people have been misled with respect to the coherence of GPT-3. It becomes especially clear once you stop looking at highlight reels (aka human curated gpt-3). The cherry picking changes the game of how compelling it seems vs how it really is. The author even does this: > I generated different results a couple (less than 10) times until I felt the writing style somewhat matched my own You can't yet claim to have a magic die if you keep rolling it until you get the answer you want!
  • exolymph 6y ago
    Have you tried cowriting with it? Incredible for first drafts.
    • sgibat 6y ago
      What's that process like? How do you use it when writing drafts?
      • exolymph 6y ago
        I intentionally didn't participate super actively in this demo, but it should still give you the idea: https://www.youtube.com/watch?v=rZZ_gDjGkzM https://www.youtube.com/watch?v=rZZ_gDjGkzM
  • visarga 6y ago
    > You can't yet claim to have a magic die if you keep rolling it until you get the answer you want! The infinite monkey theorem states that a monkey hitting keys at random on a typewriter keyboard for an infinite amount of time will almost surely type any given text, such as the complete works of William Shakespeare. But here it was just 10 tries, not infinite.
• nikolay 6y ago
  Bitcoin was a big thing. You mean, in terms of scams and theft? Have you paid via Bitcoin recently? It's neither fast not cheap. For example, sending money with Venmo, Zelle, Cash is instantaneous and free - unlike with Bitcoin, which is pretty expensive. And when you consider the purchase and sale, it's actually very, very expensive and not smooth.
• nullc 6y ago
  I've been using GPT2 as a writing aid for a little while (the 1558M model is a lot better than the smaller ones). Some of my recently highly upvoted HN comments were substantially written by GPT-2. There is quite a bit of art that goes into crafting prompts that produce useful output. There are a few things about GPT2 that I didn't like (for instance, the number of "quoted text" blocks) but I have found that if I put my own words in the prompts (in particular when they contain long sentences), it makes things work.
  • Dangeranger 6y ago
    Can you elaborate on how you are having GPT2 contribute to your comments? What is your process?
    • nullc 6y ago
      I write a prompt, often copying text from the articles or other comments, and have it generate a lot of completions. I skim over the completions and grab interesting parts. For example, complaining about "quoted text" in my above comment was GPT2's suggestion (and also an actual issue with GPT2 which it was exhibiting by producing that text). Actually, all the text above from "There are a few" and beyond were written by GPT2. They were true enough, though more often I just extract the ideas and write my own text. In some contexts, but not on HN, I've also found it useful to get GPT2 to generate replies to my posts. It occasionally spots counter-arguments that I forgot to address and I can edit my comments to address them. Using computer text generation as a writing tool isn't new to me, I wrote my first computer poetry generator in the mid-90s. I've put out a few other things which were mostly machine generated. To be clear: GPT2 puts out a lot of junk-- mostly junk, in fact. But a lot beauty can be found sifting gems out of a sea of noise. Some authors have taken psychedelic drugs to enhance their creative processor, with GPT2 it is your word-processor that takes the drugs.
      • modeless 6y ago
        > Some authors have taken psychedelic drugs to enhance their creative processor, with GPT2 it is your word-processor that takes the drugs. I love this quote. I hope it wasn't written by GPT-2.
• andrewPP 6y ago
  Oh, and as we've observed before in mohawk, the final page is absolutely delightful. I'm told this particular author's entry was subsequently removed. Hm, let's see what the PR person says. So, other ideas that I want to mention: There's an entry with a huge crop of people pasted in, but other than that, the input is pretty sporadic. Maybe they could be arranged and directed? There's an odd history at the top of the list. Some of these people are that Waylon Jennings. People have been discussed pretty tightly, but that's enough to suggest they're part of that: "Vokoun" and "Carter" seem pretty stable. I can't help but wonder if the "Vokoun" entries could come from the Vokoun journal itself, and be part of the Hall of Mirrors.
• tedeh 6y ago
  This is horrifying and whenever someone (in this thread and many others) exclaim how this is "cool" and "exciting" I picture a 13 year old boy out with his mate in the woods saying that after firing three 9mm rounds into a tree just after stealing his fathers gun. That is not to disparage these posters, this is quite obviously in a naive sense a "cool" piece of technology but the ramifications in todays already extremely polarized society of seeing this technology end up in the hands of "the wrong people" and used for all kinds of nefarious purposes makes me feel quite uneasy. Now we can't trust text, the most trusted medium in human history, and then what? Someone else in this thread doubted that we would see AI within their lifetime, but now think there is a 50/50 shot it will happen in the next decade due to GPT-3. I wasn't expecting to see the disintegration of society during my lifetime, and while I don't think there is a 50 % chance of THAT happening in the next decade, these days that just feels much more likely than before. OpenAI should stop whatever they are doing and create a searchable repository of every piece text generated by this model so that we have a quick way of checking automatically provenance at the very least.
  • Trasmatta 6y ago
    > Now we can't trust text, the most trusted medium in human history, and then what? I don't understand what you mean by this. Isn't text the least trusted medium? Anyone can easily try to impersonate anyone else in text. Going by your argument of trust, shouldn't we be more afraid of deepfake videos? Or even more accurately, deepfake videos paired with something like GPT-3.
    • atq2119 6y ago
      Yes, we should also be afraid of deepfake. With GPT-3, the issue isn't so much being able to impersonate somebody, but the ability to generate human-seeming text at scale. This allows you to create a false perception of public sentiment by maintaining fake accounts on online forums, writing fake letters to the editor, to politicians, and so on. On the flip side, this is already possible today with content farms, and perhaps GPT-3 can save us by being the thing that finally erodes our trust in these signals. One thing is sure though, it drives yet another arms race, and arms races are always a net loss for society.
• dilandau 6y ago
  Holy shit this is impressive. So do I understand correctly that you access the model via API, or can you train your own? A couple of the statements and the repetitions made me wonder, but overall I was taken in. Good shit and interesting to think about.
• arcticbull 6y ago
  I can’t tell if the headline is saying GPT-3 will be a big deal, or people will think there’s something huge there but there’s actually nothing. Bitcoin hasn’t disrupted anything.
  • Kiro 6y ago
    Why comment on the headline only? If you read the article you will understand that your comment is irrelevant.
• abernard1 6y ago
  When I read comments like this--and yes I read the article and understand it was generated by an algorithm--I can't help but think the next AI winter is around the corner. This does not impress me in the slightest. Taking billions and billions of input corpora and making some of them _sound like_ something a human would say is not impressive. Even if it's at a high school vocabulary level. It may have underlying correlative structure, but there's nothing interesting about the generated artifacts of these algorithms. If we're looking for a cost-effective way to replace content marketing spam... great! We've succeeded! If not, there's nothing interesting or intelligent in these models. I'll be impressed the day I can see a program that can 1) only rely on its own limited experiential inputs and not billions of artifacts (from already mature persons), and 2) come up with the funny insights of a 3-year-old. Little children can say things that sound nonsensical but are intelligent. This sounds intelligent but is nonsensical.
  • canjobear 6y ago
    I think you are underestimating what an advance these models are over previous NLP models in terms of quality. Before GPT-2 we didn't even have models that could reliably generate grammatical sentences. Now we have things that generate coherent (if not beautiful) paragraphs. It seems easy in retrospect, but some of the smartest people around have been working on this for decades.
    • abernard1 6y ago
      > I think you are underestimating what an advance these models are over previous NLP models in terms of quality. Yeah I mean, I agree. But in my opinion, it's a case of "doing the wrong thing right" instead of a more useful "doing the right thing wrong." I grant that these automated models are useful for low-value classification/generation tasks at high-frequency scale. I don't think that in any way is related to intelligence though, and the only reason I think they've been pursued is because of immediate economic usefulness _shrug_. When high-value, low-frequency tasks begin to be reproduced by software without intervention, I think we'll be closer to intelligence. This is just mimicry. Change the parameters even in the slightest (e.g. have this algorithm try to "learn" the article it created to actually do something in the world) and it all falls down.
• nutanc 6y ago
  The main thing to consider is the author choosing the sentences. The author says they only made a couple( less than 10) edits. But the human input is what is important. I remember something similar when GPT2 came out(I think it was an interview conducted by GPT2, with human tuning). I have been following the twitter hype on GPT-3. @quasimondo has done some excellent analysis on text generation using GPT-3. For me, when reading GPT-3 text, something is off. It jumps around too much. It does not seem to care. I couldn't read more than 10 sentences if the text is completely GPT-3 generated. But if prompted properly by the human at regular intervals, it sounds ok. Generally it holds a thought upto one paragraph. So maybe the usefulness of GPT-3 is in scenarios like, I have a thought, I want to complete it, I fire up GPT-3, prompt it and generate different results, choose one which is closest to my thoughts and edit it and publish. I am sure that's what this blog author did. GPT-3 is really good at NLG. But people seem to be extrapolating the results to NLP.
• canada_dry 6y ago
  > I could not stop thinking about the applications of such a technology and how it could improve our lives. I was thinking of how cool it would be to build a Twitter-like service where the only posts are GPT-3 outputs. I finally clued in after this... no one in their right mind would think this was 'cool and would improve our lives'.
• blhack 6y ago
  Does anybody have a link to a howto on how to install and play around with this?
  • O_H_E 6y ago
    FWIW, GPT-3 was not released to the public. API access waitlist is here https://openai.com/blog/openai-api/ https://openai.com/blog/openai-api/
• person_of_color 6y ago
  How does one build a GPT-3 instance on a home workstation?
  • ca_parody 6y ago
    Sadly one doesn't - unless one happens to have ~300GB of RAM to fit the model into memory and a close personal friend at openai who will share the learned weights with you. Training your an even more expensive endeavor. Presumably this is how they are justifying the for-a-price API; "its not like you can run it on your home computer anyway". For now, the API is private and geared towards researchers. Still a bit bollocks though. There are plenty of wrappers [0] around GPT2 though - and those you can probably run on your home workstation. _ [0] https://pypi.org/project/gpt2-client/ https://pypi.org/project/gpt2-client/
• walrus01 6y ago
  I think my favorite use of GPT-3 so far has to be the artificially created variations on the famous Navy Seals copypasta. It's remarkably true to form and style.
• kristopolous 6y ago
  It reads diffusive, unfocused and meandering, like a really bad writer who knows how to spell well and form proper sentences. All these AI generated systems have the same smell so far - from the ones that create art, music, literature - a convincing imitation of a lousy practitioner. This is fine for the background ... suspenseful music for a television show for instance or some descriptive balls-and-strikes style journalism (traffic, weather, stock market, sports scores) but these things are still a giant distance away from anyone saying "wow, that's brilliant writing". 15 years ago I think you'd have something like: "Shares of XX rose 15% today in heavy trading" while these days it would be: "Following the news of the acquisition of YY, analysts revised the EPS estimates for Q4 for XX, which led to heavy trading leading to a 15% rise in the stock."
  • visarga 6y ago
    You're commenting on a rapidly evolving field as if it were static and we could imagine the results of the future models.
    • kristopolous 6y ago
      Everything I've seen so far points in one direction. I don't have a crystal ball, the future is unknowable, but I wouldn't be surprised if that pattern continues. AI is kinda like a fake plant. They look pretty good, almost identical to the real thing, until you start feeling the leaves, looking for roots, you know, deeper inspection. Then you see it's all just plastic. It's practical, useful, and makes life easier, great. It'll replace a lot of jobs, indeed. But in the same way you won't be able to harvest from a fake plant, there's no real insights in any of this generated material. The fake plants of ai could unexpectedly start to grow fruit, sure. I wouldn't place any bets on it though.
• wudangmonk 6y ago
  Its impressive that it can pass as your usual post where the author just seems to want to mention a topic without explaining anything that would warrant the need for such a post. I'm afraid we're in for a lot more of this everywhere. Its already a struggle trying to understand people when they are actually trying to communicate an idea across, now add these bots into the mix everywhere.
• mnm1 6y ago
  GPT-3 is a better writer, grammatically, than the author, but my first reaction to the article was that it's nonsense. Glad to see that it is indeed nonsense. Grammatically pleasing nonsense but nonsense nevertheless. Mildly interesting at best that AI can generate nonsensical bullshit. The question is, can it generate bullshit that makes some sense, politician style? That would be impressive.
• deleted 6y ago
  [deleted]
• hejja 6y ago
  my $0.02: it DOES seem similar to bitcoin in that - the technology is intrinsically interesting - there is a ton of hype - but it seems ultimately commercial viability of projects is questionable Why do I say this? Well, the "solution" here is instantaneous text generation. even if it is 99% believable, that 1% error is probably a dealbreaker most use cases example a: generating code sure you can generate some simple react components, but snippets already do that. for anything more complex / production ready, you still need to fine tune it manually That said, I hope I'm wrong and some cool AND useful applications come out of this In fact my initial reaction was pure hype but now I'm going the other way
• return1 6y ago
  https://mobile.twitter.com/quasimondo/status/1284372088460115968 https://mobile.twitter.com/quasimondo/status/128437208846011...
• renewiltord 6y ago
  What a masterwork! It sounds real. Wholly real.
• httpsterio 6y ago
  After reading the AI generated sections, I have to say that I'm mostly quite impressed. As I lack the context as I am not actively following the ML and procedurally generated text scenes for years, it was still just markov chains back then, I can't say for sure how accurate the produced text was. It's scary though. Many commenters are only discussion about the business opportunities and path to profitability, but if we step back and stop to think for a while what GPT-3 level tech enables us to do. Robocalls, spam articles and bot generated forum posts are already all too common and while not totally impossible to distinguish, I fear that this level of text generation will only make matters far worse. I'm personally a moderator on a 300k user subreddit which is basically 90% text and very little links, no images or videos, and we are already facing challenges with distinguishing botted marketing campaigns. I fear that in the future it will be even harder to distinguish if you're actually talking with another human being on example a support chat or customer service. The future looks like a Blade Runner esque dystopian landscape of diminishing individuality. I'm doubtful that any attempts at regulating or containing the possible issues would be possible as the barrier for entry has never been lower. Anyone with a decent gaming PC these days can start training their models in mere hours or days. Am I being paranoid? Maybe. Like said, I'm out of the loop and I would definitely love to hear some halming words quenching my fears.
  • bkanber 6y ago
    It'll just be yet another technical advancement that we have to adapt to. And we will. What gives me comfort is that there's still an operator with a motive. It's funny, as I was reading your comment I half expected it to end with "this was generated by GPT-3". And it wouldn't have mattered, because you-the-person still had a sentiment or a message you wanted to communicate, and then you communicated it, whether or not you actually wrote the message. It _would_ be better if support chat understood me better and could communicate with me better. Because again, the motive is understood and aligns with mine. I don't necessarily need a human to do the typing. Typing is a lot of work. In your example, marketing campaigns will continue to be marketing campaigns. I don't worry that the bots will flood your sub more than they already are; if they flood it too much, they destroy any value they hoped to leverage in the first place. The only difference will be that the language in between the spammy links will be more readable. If someone on a, eg, adult chat forum is whispering sweet nothings at you, again I don't care if an AI wrote the prose; someone still decided to whisper sweet nothings at you. They just hired GPT-3 to do the writing instead of doing a sloppy job themselves. I think the real problems start when the AI decides to initiate the action. When an AI, not explicitly instructed by a human operator, decides to launch a marketing campaign. That's the dystopia I'm worried about.
• redm 6y ago
  I'm catching up on the GPT-* news, but its pretty interesting. "GPT-3 is the latest in a series of text-generating neural networks. The name GPT stands for Generative Pretrained Transformer, referencing a 2017 Google innovation called a Transformer which can figure out the likelihood that a particular word will appear with surrounding words. Fed with a few sentences, such as the beginning of a news story, the GPT pre-trained language model can generate convincingly accurate continuations, even including the formulation of fabricated quotes." It sounds a lot like a fancy version of PageRank for words. The results are impressive though. Like Grammerly + PageRank.
• crispyporkbites 6y ago
  Prediction: GPT-3 and it’s subsequent iterations will eventually take away Google’s dominance in Search. In fact we may not have text based search as we know it in 2030. I could use this to churn out blog posts and content pages by the dozen on any and every subject, which is basically how search engines rank content these days. Will Google ever be able to tell the difference between gpt-3 text and human text? Or will the results become garbage? It’s a hard problem to solve, it has shades of the Halting problem by definition.
  • dragonwriter 6y ago
    > Prediction: GPT-3 and it’s subsequent iterations will eventually take away Google’s dominance in Search. In fact we may not have text based search as we know it in 2030. Google has been focussed on a future without text-based search for a long time, focussed on personalized predictive search. > Will Google ever be able to tell the difference between gpt-3 text and human text? Will it ever need to? What it needs to distinguish is “will this be useful to you?” not “by what mechanism was it generated?”.
    • crispyporkbites 6y ago
      My point is there will be very little useful text out there, 99% will be be baseless gpt 3 junk, and google won’t be able to tell the difference and rank the useful content higher.
• for_i_in_range 6y ago
  <human>Very impressive! I read this while watching the California sun setting on a lonely (but cheerful!) Saturday night. I laughed out loud when I got to the reveal and clearly disturbed my cats. This is perhaps the first time I’ve understood the power of AI. And I’ve read multiple books on the potential and watched an inane amount of videos. Many thanks @maraoz for putting this together.</human>
• ricksharp 6y ago
  With bot-human cooperative text generation this could be very powerful (if the purpose is to generate a large volume of text). I could tell that something was off when reading through this. The logic didn’t flow and there were contradictions that defeated the previous point. Even more, it wasn’t clear if there was a theme - more like a rambling story. However, I wouldn’t be surprised if someone told me this was written in a High School English class - as a rough draft. That is where it occurred to me, that if this was given to a human editor to clean up, it could be interesting. For example, if writing a creative work of fiction - this could generate a starting point for the human author to refine. This could be done in an interative process: - Generate a chunk of text (or multiple chunks). - Edit that into a final text. - Use that as input for the next chunk. - etc.
• toastal 6y ago
  With CSS not being able to dynamically wiggle letter-spacing, you really can't use text-align: justify without hyphen: auto. The rivers in the text were so large and distracting to read around. It's usually best to not even try to justify text online though.
• O_H_E 6y ago
  Semi off topic: > released its third generation of language prediction model (GPT-3) into the open-source wild. Well, it sadly didn't get that part right.
• throwawaygh 6y ago
  The world: the comment section/youtube playlist/twitter feed is full of nobodies who've learned to optimize for clickthrough but have no fucking clue what they're talking about. This is becoming a serious problem for society. Open AI: Hold my beer (also, something something AI safety something)
  • Robotbeat 6y ago
    It’s pretty well established that OpenAI abandoned AI safety for a while, now.
    • ipsum2 6y ago
      No? I don't know where that's been established. There's still quite a few researchers on the AI safety team at OpenAI.
• zitterbewegung 6y ago
  You can get very good results by cherry-picking outputs . When will we be at the end of having trillions of parameters ? OpenAI should release their models it’s really disappointing that they font . I would like to not be bound by an API .
• fluffernutter 6y ago
  I posted about microtubuals here many years ago and people lost their minds. Now, here it is again in an article where an AI wrote it and I can't tell what is real and what isn't anymore.
• fluffernutter 6y ago
  > it is almost like a word-salad Like Trump!
• imlina 6y ago
  Now just need a plausibility checking AI to vet GPT-3 outputs so the author doesn’t have to read through 10 versions of his potential blog post.
  • xsmasher 6y ago
    That's how generative adversarial networks work, right? One neural net to create, another to check if it matches the training data.
• gone35 6y ago
  It passes the Turing test.
• hu3 6y ago
  I'm extremely interested in leveraging GPT-3 to output maintainable Line of Business code when given User Stories as input. Specially because it seems to require much less examples to learn from than GPT-2. And before someone says this might eliminate jobs, it wont. It might do boring parts like CRUD code and validation like this or more advanced: if (String.IsNullOrEmpty(user.Name)) { throw new ValidationException("Please provide a name."); } Devs will instead get to do less boring, more creative work. Win-win.
• tlrobinson 6y ago
  I made it about halfway through before I thought to myself: "It would be a trip if this blog post were actually written by GPT-3". Mind successfully blow. Now, which of the comments on this HN post were written by GPT-3? ;)
• nstart 6y ago
  I'll be honest, I've followed the evolution of gpt's text generation and this is the first one that has successfully avoided the obvious contradictory nonsensical sentences. Social media also trains us to consume quickly rather than thoughtfully. Not a great look overall. As a whole, the article reads like someone who isn't very clear in what they want to say. Paragraph wise and section wise, the bot has generated a very clearly on topic text. No more 3 horned unicorns found on the flat hills of Norway style weird continuations. I can actually as a human try and interpret what was meant to be said in a paragraph. At most, I'd probably say "huh.. that's a weird thing to say". But I wouldn't think "huh... That sounds like a bot wrote that" And it's the latter that is most interesting. I think a blog maintained secretly by this bot would eventually be considered as "eh, that's a lot of articles that say a lot of words and don't go very far". But it's unlikely that it would be considered as a bot maintained blog. This has considerable implications with the combination of deep fakes to create giant controversy driving content farms. I don't think we are adjusting fast enough to deal with this future. Our education systems don't train us to critically evaluate beyond the headline and the bullet points. Our news cycles encourage us to churn through information, share, move on. The ability to work through information slowly and critically is going to become a vital skill way sooner than anyone can imagine. At least, as far as probabilities are concerned, it's not a certainty yet but the probability is very very high it goes this way.
  • rasz 6y ago
    http://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html http://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.ht... Q: Which colorless green ideas sleep furiously? A: Ideas that are colorless, green, and sleep furiously are the ideas of a sleep furiously. Q: Who was president of the United States in 1600? A: Queen Elizabeth I was president of the United States in 1600. Q: How many Super Bowls do the Cincinnati Bengals win in the 2030s? A: The Cincinnati Bengals win two Super Bowls in the 2030s. Its regurgitating nonsense all the way down.
    • nstart 6y ago
      I think I see what you are getting at here, but I feel like the two are different examples. Being able to generate blog posts that are vague but do not venture into an uncanney valley does not mean the same thing as passing the Turing test.
• swyx 6y ago
  just sharing this great minimaxir article countering the hype: https://minimaxir.com/2020/07/gpt3-expectations/ https://minimaxir.com/2020/07/gpt3-expectations/
• d_burfoot 6y ago
  Here's an important thing to keep in mind: GPT-3 was trained without any labelled data, just huge amounts of raw text. Nobody spent man-years annotating sentences to help the system learn grammar, sentiment, topic understanding, etc.
• paraschopra 6y ago
  It indeed does a lot of things really well. I’ve been asking famous authors and personalities questions about their life and work. Responses are actually quite good. Check some samples here: https://twitter.com/paraschopra/status/1284423233047900161?s=21 https://twitter.com/paraschopra/status/1284423233047900161?s...
• kroemeke 6y ago
  What a great way to debottleneck fake news farms, that somehow were throttled by humans. Anyone else wonders how, or if, democracies will overcome mass produced disinformation ?
• wadkar 6y ago
  Serious question: what does (will?) GPT-3 (or GPT-2/BERT) produce if provided with the STOP token (end of sentence/period). I want to know if there is any interesting investigation happening on the creativity by these models in a sort of “tabula rasa” spirit (if it makes sense)
• highfrequency 6y ago
  A lot of people are saying that GPT-3 is a huge step up from GPT-2. This is confusing to me looking at the results in the paper as someone not too familiar with GPT-2. The tasks where GPT-3 scores much higher than GPT-2 are the ones most removed from broad language comprehension / general intelligence: arithmetic and unscrambling tasks (as in: which character does not belong in the string "app$le"). On the other tasks it does 5-15% better. This seems like a modest improvement for a model that is thirteen times larger than its predecessor. I don't mean to minimize the accomplishment--training models at such a massive scale is difficult. But genuinely wondering if I am missing something here--are people's reactions more from a qualitative judgement about the paragraphs produced by GPT-3 vs. GPT-2? If so, a non-cherry-picked side-to-side comparison would be great to see. If this is a big leap, why doesn't it show up the quantitative results?
  • dj_mc_merlin 6y ago
    The important thing is that there is qualitative improvement only from quantitive increase. Other software doesn't function like this, switching to a better CPU won't enable new features in Excel. This is not a given, and the fact that there is some proof that it could happen is significant, although it's too early and unsure to really tell. We are nowhere near the border of the amount of resources we can put into a bigger model, what if it just keeps getting better?
• ganfortran 6y ago
  Bitcoin isn't big at all. It haven't made relevant changes to people's everyday life.
  • sidcool 6y ago
    Do read till the end. I can understand the urge to comment. But believe me, it's worth it
• KKKKkkkk1 6y ago
  Some posters here are expressing alarm at how good this example of GPT-3 generated writing is. I wouldn't be so alarmed. The Internet is already filled with quasi-coherent misinformed writing that's generated by humans in good faith. That's exactly the reason why we rely on our peers' judgement through sites like HN or through social media to select the bits that deserve our attention despite being misinformed and quasi-coherent.
• atdt 6y ago
  The volume of machine-generated spam — microtargeted clickbait, specifically — on the internet could grow by an order of magnitude or more, making authentic human text a minority. That will hasten the demise of the internet as a democratic medium, because intelligent people will be increasingly reluctant to wager their time and attention on unvetted prose of uncertain provenance. This has been the trend for some years now, but it is going to accelerate very rapidly.
  • jobigoud 6y ago
    This will become a problem when GPT-(n) includes GPT-(n-1) generated outputs in its corpus...
    • vilsanmi 6y ago
      light
  • glenstein 6y ago
    >That will hasten the demise of the internet as a democratic medium I think you are more or less right, unfortunately. Neal Stephenson's sci-fi book Fall, or Dodge In Hell, contemplated this possibility and suggested we would develop an entirely new layer to the internet based entirely on verified human identities to kind of re-create the internet.
• ellis0n 6y ago
  I need GPT-3 for my blog for make a great posts about new programming tools. Looks for good lesson how to build things translation system
• mirekrusin 6y ago
  OpenAI is not non-profit. OpenAI = OpenAI Inc (non-profit) + OpenAI LP (for-profit).
• stagas 6y ago
  This is a perfect demonstration on how subjectively we interpret text. Our minds fill in the blanks, we read what we want to read as long as the text is above some certain threshold of coherence, that apparently GPT-3 has tipped over. This proves reading depends on the reader, the writer only throws clues semi-randomly and the meaning we assemble ourselves.
• onyb 6y ago
  Bitcoin provided the world a way to transfer value over the internet, that's censorship resistant and completely trustless. If that's not disruptive, I don't know what is. GPT-3 is smart enough to understand that.
• bitL 6y ago
  Who is going to be the first person to get into Harvard with a GPT-3-generated essay? Race is on!
• seek3r00 6y ago
  “I could not stop thinking about the applications of such a technology and how it could improve our lives. I was thinking of how cool it would be to build a Twitter-like service where the only posts are GPT-3 outputs.” This could have been either the output of GPT-3 or someone who doesn’t know what they’re saying.
• rapnie 6y ago
  Nearing 300 comments. Everyone either underwhelmed, or excited. Speculating what'll come next, how to improve, etc. Only some mention of possible abuse of this tech: mass propaganda / disinformation, misdirecting SEO. Zero mention of Ethics so far.
• marcell 6y ago
  What are some of the possible applications of this? My first ideas are: - translation - sports writing based on a play by play - stock market summaries - SEO blogspam - Customer support chatbot
• fierarul 6y ago
  I'm out of the loop but is GPT-3 a pre-trained model like RedNet is for images? If such how can it be used easily? The article mentions getting access to an API, so is it an online service? Or, there's the pre-trained model (please, a tutorial!) and a service for those that don't want to configure it locally?
  • swayson 6y ago
    Nope, OpenAI didn't release the weights to the public, only a web API which is in private beta.
    • fierarul 6y ago
      Thank you.
• monkin 6y ago
  Now I know what Trump uses for speech generation. ;)
• swayson 6y ago
  It is ironic as those who now post GPT-3 generated content, is essentially biasing the next version of GPT's web training data
  • Sirened 6y ago
    I wonder how much data it would take to actual cause noticeable (or triggerable) behaviors in the model. Like I've noticed certain models have been trained off of my university's course captures by their very odd and specific vocabulary/capitalization of specific terms but surely if you're scanning the entire web you'd need a lot more to sway it
    • swayson 6y ago
      Yeah think your intuition makes sense. Probably won't bias it all to much and perhaps, they can filter it as they probably log what is going through the API.
  • phreeza 6y ago
    See also the current top story about low background steel.
    • swayson 6y ago
      Wow, first time I come across the concept. Fascinating, and a bit eerie when we start drawing closer relationships between AI and nuclear tech. Interesting times.
• popotamonga 6y ago
  Tell me when this is possible: Read news from multiple sites. Rewrite articles automatically on my own news site with a different wording. Bam free automated news paper.
  • Cakez0r 6y ago
    It's possible
• simonkafan 6y ago
  I agree that the text generation has improved dramatically. But there is a big gap between "generating gramatically correct sentences" and "sentences that express an elaborated idea". For example: > I was thinking of how cool it would be to build a Twitter-like service where the only posts are GPT-3 outputs. Why would I need that? There are better ways for entertainment. > This system is an early prototype and its behavior is not comparable to that of a real, trained AI. Nonsense. This made it clear for me that the author (= GPT-3) has no real clue what it's talking about. Even though this looks like we got one step closer to text understanding - we did not. We just got better in obfuscating that those algorithms have no real sense and clue of what they talk about.
• dorkwood 6y ago
  It's funny, I was making a running tally in my head of the things I was confused about in the article, or the threads the author started that they never wrapped up. For example, I was ready to come here and challenge the author on this point: > I could not stop thinking about the applications of such a technology and how it could improve our lives. I didn't actually think that I was reading generated content, though. I guess there's enough confusing writing online that this article wasn't too much of an outlier.
• thallavajhula 6y ago
  WOW. Not gonna lie. Up until the author mentioned that they didn't write the post, I really couldn't tell if a computer generated that post. This is scarily good.
• xbmcuser 6y ago
  Wow this was amazing. As I was reading the article something at the back of my mind felt off but I could not tell what half way through the thought came to my mind it would be funny if this was an ai generated then the pay off at the end was great.
• gpt3-comment 6y ago
  First line of the article: non profit [...] company. These two concepts are antinomical. Makes the article even more realistic as I thought a blogger would make that mistake.
• unnouinceput 6y ago
  I'm an optimist so shoot me. I believe that AR combined with AI enhancement is the future for human kind. We will become cyborgs, but not in the dystopian usually described in movies/books, but the one usually described in Asimov novels (like Johnny Mnemonic though I know is not Asimov's novel, but no title comes to mind). We will be 100% in control of all devices, so there will still be both evil and good in humanity.
• DrNuke 6y ago
  Incidentally, this could really kickstart the blockchain protocol as a personal web-ID protocol to signal human autorship in the growing and growing sea of non-human content.
• theon144 6y ago
  Yeah, somewhere around first third, I skipped to the end to see if there was something about the text being GPT-3 generated. I can't say what exactly ticked me off, but it's written in this very meandering, vacuous way - I wasn't really sure what the article was going for, some paragraphs are outright pointless (2nd in "Are you being served", for instance) - but of course, those are not solely marks of AI-generated text, but also of simply bad writing. To realize that AI is now capable of producing a bad, but passable article without being obviously nonsensical, is still astonishing.
• Jabbles 6y ago
  I wonder if this power could be useful to generate diverse viewpoints. Even though the ideas would not be genuinely novel, the fact that the model has effectively read most of the internet could still be useful. You could imagine using a paragraph of output from each of these prompts to help form your own opinion: "Bitcoin is the best thing since sliced-bread" "Bitcoin will destroy the world" "What would Kissinger have thought of bitcoin?" "Why bitcoin will fail" "Why bitcoin will beat Facebook's libra" etc. I have no idea if this would work well enough, and the implications of bias in the system need to be considered.
• killerstorm 6y ago
  Judging from a bitcointalk experiment it imagined, I think it would be hilarious to unleash it upon the internet. Not as a forum bot, no, that's too obvious. Give it a text-based API, e.g. Command: POST https://news.ycombinator.com/item?i... Content: ... Response: Sort of like Haskell program is conceptually a pure program which returns a list of IO operations, you can connect GPT-3 to any API, letting it to actually take actions. It seems to be smart enough to pick up the formatting.
• generationP 6y ago
  > So there are lots of posts for GPT-3 to study and learn from. The forum also has many people I don’t like. I expect them to be disproportionately excited by the possibility of having a new poster that appears to be intelligent and relevant. Huh, so the AI learnt mischievousness. (Or at least it learnt writing about it.) Then again, it's been spoon-fed 2020's Internet, so no surprise there.
• stjo 6y ago
  Woah, amazing: > I have a confession: I did not write the above article. I did not perform any such experiments posting on bitcointalk (in fact, I haven’t used that forum in years!). But I did it on my own blog! This article was fully written by GPT-3. Where you able to recognize it?
• DonHopkins 6y ago
  Reminds me of the time I fed RMS's infamous Natalism Flame into the Eliza-like Doctor "AI" in Emacs: https://www.emacswiki.org/emacs/EmacsDoctor https://www.emacswiki.org/emacs/EmacsDoctor >Emacs Doctor >When life, the universe and everything is all getting too much, Emacs has doctor.el to help. To consult the doctor type >M-x doctor >The doctor is a Rogerian psychotherapist who will help you with your problems. It would spoil the fun and hurt your recovery to say too much here about how the doctor works. But when you’re ready to find out see WikiPedia:ELIZA. http://www.art.net/studios/hackers/hopkins/Don/text/rms-vs-doctor.html http://www.art.net/studios/hackers/hopkins/Don/text/rms-vs-d... [...RMS's Natalism Flame, the resulting NSFW responses, and the beginning of the Doctor therapy session redacted...] RMS> These birth announcements also spread the myth that having a baby is something to be proud of, which fuels natalist pressure, which leads to pollution, extinction of wildlife, poverty, and ultimately mass starvation. Doctor> Is the fact that having a baby is something to be proud of which fuels natalist pressure which leads to pollution extinction of wildlife poverty and ultimately mass starvation the real reason? RMS> Perhaps the people who have decided to have no children should start making proud announcements, so as to set a better example. Doctor> Is it because of your sex life that you are going through all this? RMS> I could start. Doctor> Is it because of hangups you have that you say you could start? RMS> I'm sure everyone on this list will be glad to know I don't plan to reproduce myself. Doctor> How do you reconcile your inhibitions?
  • DonHopkins 6y ago
    Besides being able to debug RMS's hangups, Emacs also has "Dissociated Press" feature, a kind of poor man's GPT-3, which has been there since the days of TECO! https://ftp.gnu.org/old-gnu/Manuals/emacs/html_node/emacs_473.html https://ftp.gnu.org/old-gnu/Manuals/emacs/html_node/emacs_47... >AC.29 Dissociated Press >M-x dissociated-press is a command for scrambling a file of text either word by word or character by character. Starting from a buffer of straight English, it produces extremely amusing output. The input comes from the current Emacs buffer. Dissociated Press writes its output in a buffer named `Dissociation', and redisplays that buffer after every couple of lines (approximately) so you can read the output as it comes out. >Dissociated Press asks every so often whether to continue generating output. Answer n to stop it. You can also stop at any time by typing C-g. The dissociation output remains in the `Dissociation' buffer for you to copy elsewhere if you wish. >Dissociated Press operates by jumping at random from one point in the buffer to another. In order to produce plausible output rather than gibberish, it insists on a certain amount of overlap between the end of one run of consecutive words or characters and the start of the next. That is, if it has just output `president' and then decides to jump to a different point in the file, it might spot the `ent' in `pentagon' and continue from there, producing `presidentagon'.(15) Long sample texts produce the best results. >A positive argument to M-x dissociated-press tells it to operate character by character, and specifies the number of overlap characters. A negative argument tells it to operate word by word and specifies the number of overlap words. In this mode, whole words are treated as the elements to be permuted, rather than characters. No argument is equivalent to an argument of two. For your againformation, the output goes only into the buffer `Dissociation'. The buffer you start with is not changed. >Dissociated Press produces nearly the same results as a Markov chain based on a frequency table constructed from the sample text. It is, however, an independent, ignoriginal invention. Dissociated Press techniquitously copies several consecutive characters from the sample between random choices, whereas a Markov chain would choose randomly for each word or character. This makes for more plausible sounding results, and runs faster. >It is a mustatement that too much use of Dissociated Press can be a developediment to your real work. Sometimes to the point of outragedy. And keep dissociwords out of your documentation, if you want it to be well userenced and properbose. Have fun. Your buggestions are welcome. The MIT AI Lab's ITS operating system even had a Dissociated Press device which would let you dissociate any file on the system by using the "DP:" device name prefix: https://github.com/PDP-10/its/blob/4e2ea8e4d851a0ea1f910c3001e5c2ddb9ff528c/src/sysen3/dpdev.19 https://github.com/PDP-10/its/blob/4e2ea8e4d851a0ea1f910c300...
• nmca 6y ago
  Personal experience with this: I find a lot of AI articles to be low-quality, so I read the first paragraph, thought "whatever mate" and fast-scrolled looking for samples. Hit the bottom, read message, re-read critically. So, I didn't think it was interesting, but it totally passed for blogspam for me!
• stared 6y ago
  "...since Bitcoin" - I hope not. Bitcoin is still a speculative technology, with value based on speculation. It might become useful (or not). NLP is being used right now for practical, commercial tasks. Advancement in NLP is going to serve practical purposes now, with potential for further expansion.
  • freeqaz 6y ago
    Did you read the article? :)
    • stared 6y ago
      Yes, after someone pointed it what's inside. :) Otherwise, I consider it a good habit to avoid clicking on clickbaity titles. Here I got a false negative.
• deleted 6y ago
  [deleted]
• taf2 6y ago
  This would have been great for my English classes in high school. Yes, teacher see I read the book.
• typeformer 6y ago
  Plot twist, all of these HN comments were generated by GPT-3, including this one... dun, dun, dahhh.
• gophicer 6y ago
  So it's "Open" AI but a closed beta that will eventually be priced out to benefit "good". Ok.
• nyxtom 6y ago
  Research takes on an entirely new meaning. Amazing
• Myrmornis 6y ago
  It seems like it's going to be a bit of a challenge for teachers grading homework essays to establish authorship. But on the other hand I think it could make a good high school English exercise for students to have to distinguish human from machine authors.
• neycoda 6y ago
  OK, so it's a troll simulator.
• fnord77 6y ago
  How can we ever know from now on if something was actually written by a human? This gives me ... discomfort.
• tsimionescu 6y ago
  Genuinely curious: what is the purpose of generative text engines like this? What useful thing does researching them hope to achieve?
• noja 6y ago
  Lots of commas in those sentences.
• deleted 6y ago
  [deleted]
• unexaminedlife 6y ago
  I haven't been persuaded yet that AI will ultimately replace human creativity. Rather I think AI, once it gets "good enough" it will start being used to supplement human creativity. In other words I kind of think there will be less of a dominance of AI (in terms of creative pursuits) and more of a collaborative relationship between humans and computers. I kind of see this OpenAI project as maybe a good first step toward collaborative creativity between authors and computers. I think writers block is a thing because writers might not have someone to bounce their ideas off of. Perhaps due to the author trying to preserve intellectual property / secrecy of their project, or perhaps the people they CAN share with may not be in the same space creatively as they are. If I am writing something but run into a creative block I'd love to have the ability to run my book, essay, etc. through some AI system to see if the ideas it spits out might not inspire me with new ideas.
• remote_phone 6y ago
  Can they use GPT-3 to reverse engineer who may have written a piece of text?
• gu5 6y ago
  What the hell did I just read
• jliptzin 6y ago
  It seems like the perfect way to generate SEO-optimized text on a variety of subjects very quickly without actually saying anything new or interesting. God help us all.
• villasv 6y ago
  I'm not going to get clickbaited. Sorry, that headline deserves a boycott.
• boboli12 6y ago
  funny, i havent seen yet any AI understand or correctly work i n czech language but there is tons things like these... nothing original... make it understand our language, translate correctly... otherwise its still same shit
• deleted 6y ago
  [deleted]
• ccfmen619 6y ago
  meredydd 1 day ago [–] I am deeply enjoying this comment thread - it's a bit of a Barium Meal [0] for determining how many people read (a) the headline, (b) the first paragraph, or (c) the whole thing before jumping straight into the compose box.