8 ms·

The thing that kills me is that to the vast majority of human beings the nonsensical technobabble above is probably indistinguishable from real, honest, logical

by cs702 6y ago

The thing that kills me is that to the vast majority of human beings the nonsensical technobabble above is probably indistinguishable from real, honest, logically consistent technobabble.[a] Soon enough, someone will replicate the Sokal hoax[b] with GPT-3 or another state-of-the-art language-generation model. It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the social sciences. [a] https://en.wikipedia.org/wiki/Technobabble https://en.wikipedia.org/wiki/Technobabble [b] https://en.wikipedia.org/wiki/Sokal_affair https://en.wikipedia.org/wiki/Sokal_affair -- here's a copy of Sokal's hoax paper, "Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity:" https://physics.nyu.edu/faculty/sokal/transgress_v2/transgress_v2_singlefile.html https://physics.nyu.edu/faculty/sokal/transgress_v2/transgre...

thewarrior 6y ago
The thing is that this thing has now crossed into the uncanny valley. Earlier it would have great trouble making a single sentence that makes sense. You only ever remember whether the last two sentences made sense and go together. And with GPT-3 any pair of sentences always makes almost perfect sense. By the time you're four sentences down you go wait a minute ...
- setzer22 6y ago
  This was very apparent when reading the generated stories [1]. Especially the shoggoth cat dialogue, I found that one really creepy. The fragment below comes straight out from the uncanny valley: Human: Those memes sound funny. But you didn’t include any puns. So tell me, what is your favorite cat pun? AI: Well, the best pun for me was the one he searched for the third time: “You didn’t eat all my fish, did you?” You see, the word “fish” can be replaced with the word “cats” to make the sentence read “Did you eat all my cats?” [1]: https://www.gwern.net/GPT-3 https://www.gwern.net/GPT-3
  - spion 6y ago
    An instance of a Voight-Kampff test.
    - techdragon 6y ago
      Yeah, GPT-3 never really gives any kind of answer to “why”. It rambles on like Abraham Simpson talking about how an onion on his belt was the style at the time. Devoid of purpose it fills the void with meaningless words until told to stop. It’s subtle gibberish... and fucking irritating as soon as you catch it.
CamperBob2 6y ago
Worse, Sokal's Revenge is probably inevitable, in which someone will generate a nightmarishly complex mathematical proof that takes the whole field in unexplored directions. Some of the most respected professors and most promising students will then be distracted for years trying, and ultimately failing, to make sense of it. Some say this has already happened. Nobody has ever seen the Social Text editors and Mochizuki in the same room together, have they?
barrkel 6y ago
In a similar way to how image detection networks appear to key largely on visual textures, GPT-3 seems to key on jargon, tone and pacing, the texture of speech within specific milieus.
thewarrior 6y ago
It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the social sciences. And then, it'll be all over for us. We won't have any more funding and our jobs will disappear. I can already hear the protests: "But we're not just scientists! We're also philosophers!" Well, yes and no. Philosophers are supposed to think about things philosophically, but they don't actually do anything about them; they're just entertainers. Scientists do something about them. They make things happen. And when those things happen, people take notice. If science and technology have a weakness, it's that they work too well. This was probably a strength at one point, but not anymore. In the not-too-distant future, there probably won't be any more philosophy professors; there will just be philosophers. But only in the same sense that there are lions and mushrooms. This comment was also written by GPT-3.
- jjoonathan 6y ago
  Somebody get this thing onto scp-wiki, it will be right at home. Not gonna lie, I went poking around to see if I could get my hands on it, but it seems like the answer is no, for now.
  - StavrosK 6y ago
    That's a good question, how do we get access? I signed up on the list, but it must be thousands of people long by now. Does anyone here know anyone who can get people access?
- 6nf 6y ago
  How many attempts did it take or did you just choose the first one? I have to admit, this is passing my turing test...
  - geofft 6y ago
    Really? I got about halfway through and realized that the comment had no point. If you tried to summarize what it was arguing, beyond the first sentence, I don't think you could make a coherent summary. Maybe the real lesson is we don't expect human-written comments on discussion fora to be particularly coherent....
p1esk 6y ago
If he didn't put in the last line ("plot twist ...") I'm pretty sure no one here on HN would have guessed it. In fact, while reading that comment I started to wonder why no one has tried to use GPT to generate text one character at a time. Or if someone has, what are the advantages and disadvantages over the BPE approach.
- foobarbecue 6y ago
  Your second paragraph is GPT-3?
  - O_H_E 6y ago
    Ok, now this is getting deep and I don't like it.
    - Scarblac 6y ago
      I assume both previous comments, and this one, are also GPT-3? Edit: it is amusing to think that soon the way to distinguish them will be that human comments have weird errors caused by smartphone keyboard "spell checking" in them...
      - p1esk 6y ago
        You really think my comment above was GPT-3 generated? Wow. Did I really make so little sense?
        deleted 6y ago
        [deleted]
        foobarbecue 6y ago
        On a re-read, I'm not sure why I thought that, sorry. Context: I don't know much about machine learning and when I was scanning through comments, doing text generation one character at a time seemed silly and I must have been in the grips of "everything could be GPT!" hysteria. My robot detector needs work, clearly. Need to get educated.
pests 6y ago
One of my favorite conspiracists, Miles Mathis, has this quality. He can strong together entire pages of very real and consistent nonsense that is totally logical and makes enough sense to be real. I have to remember I'm not reading a legit theory and now really do confuse myself with his version of science vs reality.
g_p 6y ago
There's actually precedent for this - sci-gen got papers accepted in IEEE and Springer publications, through peer review, and they had to investigate their back catalogue to look for others. Given the propensity for academic writing to often favour the strategy of confusing the author through obfuscation (to make a minor advance sound more significant than it is), I suspect tools like this could, as you say, actually get published papers in some fields like social sciences. In an engineering or science paper you can check equations match conclusions, and that graphs match data etc. In a more qualitative field of work, reviewed in a publish-or-perish system that doesn't incentivise time spent on detailed reviewing, I think there's a very real risk babble like this just comes across like every other paper they "review". I think it takes a certain level of confidence to dismiss others' work as nonsensical waffle, but sadly this is a confidence many lack, and they assume there must be some sense located therein. Marketing text is a great place to train yourself to recognise much of what is written is meaningless hocum. Sci-Gen - https://pdos.csail.mit.edu/archive/scigen/ https://pdos.csail.mit.edu/archive/scigen/ Reporting on withdrawals of papers - https://www.researchgate.net/publication/278619529_Detection_of_computer_generated_papers_in_scientific_literature https://www.researchgate.net/publication/278619529_Detection...
- 29athrowaway 6y ago
  Dr. Herbert Schlangemann (fake alias for Sci-gen) not only got papers accepted in journals, it was invited to participate as session chair at conference co-sponsored by IEEE. https://en.wikipedia.org/wiki/SCIgen#Schlangemann https://en.wikipedia.org/wiki/SCIgen#Schlangemann
pasabagi 6y ago
Kind of annoying 'hoax', though. Obviously you can publish garbage in fringe journals if you leverage pre-existing prestige and position like Sokal did. Doesn't really say anything about the social sciences. You can also publish a lot of nonsense in certain chinese journals that optimize for quantity in quality, in whatever field you want.
yaj54 6y ago
The thing that kills me is that soon enough, a "fake" paper written by GPT-3 will get published in an academic journal because it has actually contributed a new insight. It's easy to consider text generation models as "just mimicking grammar". But isn't grammar also just a model of human cognition? Is GPT modeling grammar or is it modeling human cognition? Since GPT can ingest radically more text (aka ideas) won't it soon be able to generate texts (aka ideas) that are a more accurate collation of current knowledge than any individual human could generate? -- [Was this comment written by GPT-3?]
- Udik 6y ago
  My impression is that these models are already doing far more than what the language production machinery in our brain does. We are able to produce language according to grammar and semantics, but we also have independent mental representations to guide the generation of language and to provide context. I don't really understand why we're trying so hard to build models that can generate coherent texts based on having predigested only other texts, without any other experience of reality. Their capabilities appear already superhuman in their ability to imitate styles and patterns of any kind (including code generation, images, etc.). It feels like we're overshooting our target by trying to solve an unsolvable problem, that of deriving the semantics of reality from pure text, without any other type of input.
- lostmsu 6y ago
  It was not - there's a point :-D I am impressed though nobody dared to guess in 2 weeks.

by cs702 6y ago

thewarrior 6y ago

The thing is that this thing has now crossed into the uncanny valley. Earlier it would have great trouble making a single sentence that makes sense. You only ever remember whether the last two sentences made sense and go together. And with GPT-3 any pair of sentences always makes almost perfect sense. By the time you're four sentences down you go wait a minute ...

setzer22 6y ago
This was very apparent when reading the generated stories [1]. Especially the shoggoth cat dialogue, I found that one really creepy. The fragment below comes straight out from the uncanny valley: Human: Those memes sound funny. But you didn’t include any puns. So tell me, what is your favorite cat pun? AI: Well, the best pun for me was the one he searched for the third time: “You didn’t eat all my fish, did you?” You see, the word “fish” can be replaced with the word “cats” to make the sentence read “Did you eat all my cats?” [1]: https://www.gwern.net/GPT-3 https://www.gwern.net/GPT-3
- spion 6y ago
  An instance of a Voight-Kampff test.
  - techdragon 6y ago
    Yeah, GPT-3 never really gives any kind of answer to “why”. It rambles on like Abraham Simpson talking about how an onion on his belt was the style at the time. Devoid of purpose it fills the void with meaningless words until told to stop. It’s subtle gibberish... and fucking irritating as soon as you catch it.

CamperBob2 6y ago

Worse, Sokal's Revenge is probably inevitable, in which someone will generate a nightmarishly complex mathematical proof that takes the whole field in unexplored directions. Some of the most respected professors and most promising students will then be distracted for years trying, and ultimately failing, to make sense of it. Some say this has already happened. Nobody has ever seen the Social Text editors and Mochizuki in the same room together, have they?

barrkel 6y ago

In a similar way to how image detection networks appear to key largely on visual textures, GPT-3 seems to key on jargon, tone and pacing, the texture of speech within specific milieus.

thewarrior 6y ago

It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the social sciences. And then, it'll be all over for us. We won't have any more funding and our jobs will disappear. I can already hear the protests: "But we're not just scientists! We're also philosophers!" Well, yes and no. Philosophers are supposed to think about things philosophically, but they don't actually do anything about them; they're just entertainers. Scientists do something about them. They make things happen. And when those things happen, people take notice. If science and technology have a weakness, it's that they work too well. This was probably a strength at one point, but not anymore. In the not-too-distant future, there probably won't be any more philosophy professors; there will just be philosophers. But only in the same sense that there are lions and mushrooms. This comment was also written by GPT-3.

jjoonathan 6y ago
Somebody get this thing onto scp-wiki, it will be right at home. Not gonna lie, I went poking around to see if I could get my hands on it, but it seems like the answer is no, for now.
- StavrosK 6y ago
  That's a good question, how do we get access? I signed up on the list, but it must be thousands of people long by now. Does anyone here know anyone who can get people access?
6nf 6y ago
How many attempts did it take or did you just choose the first one? I have to admit, this is passing my turing test...
- geofft 6y ago
  Really? I got about halfway through and realized that the comment had no point. If you tried to summarize what it was arguing, beyond the first sentence, I don't think you could make a coherent summary. Maybe the real lesson is we don't expect human-written comments on discussion fora to be particularly coherent....

p1esk 6y ago

If he didn't put in the last line ("plot twist ...") I'm pretty sure no one here on HN would have guessed it. In fact, while reading that comment I started to wonder why no one has tried to use GPT to generate text one character at a time. Or if someone has, what are the advantages and disadvantages over the BPE approach.

foobarbecue 6y ago
Your second paragraph is GPT-3?
- O_H_E 6y ago
  Ok, now this is getting deep and I don't like it.
  - Scarblac 6y ago
    I assume both previous comments, and this one, are also GPT-3? Edit: it is amusing to think that soon the way to distinguish them will be that human comments have weird errors caused by smartphone keyboard "spell checking" in them...
    - p1esk 6y ago
      You really think my comment above was GPT-3 generated? Wow. Did I really make so little sense?
      - deleted 6y ago
        [deleted]
      - foobarbecue 6y ago
        On a re-read, I'm not sure why I thought that, sorry. Context: I don't know much about machine learning and when I was scanning through comments, doing text generation one character at a time seemed silly and I must have been in the grips of "everything could be GPT!" hysteria. My robot detector needs work, clearly. Need to get educated.

pests 6y ago

One of my favorite conspiracists, Miles Mathis, has this quality. He can strong together entire pages of very real and consistent nonsense that is totally logical and makes enough sense to be real. I have to remember I'm not reading a legit theory and now really do confuse myself with his version of science vs reality.

g_p 6y ago

There's actually precedent for this - sci-gen got papers accepted in IEEE and Springer publications, through peer review, and they had to investigate their back catalogue to look for others. Given the propensity for academic writing to often favour the strategy of confusing the author through obfuscation (to make a minor advance sound more significant than it is), I suspect tools like this could, as you say, actually get published papers in some fields like social sciences. In an engineering or science paper you can check equations match conclusions, and that graphs match data etc. In a more qualitative field of work, reviewed in a publish-or-perish system that doesn't incentivise time spent on detailed reviewing, I think there's a very real risk babble like this just comes across like every other paper they "review". I think it takes a certain level of confidence to dismiss others' work as nonsensical waffle, but sadly this is a confidence many lack, and they assume there must be some sense located therein. Marketing text is a great place to train yourself to recognise much of what is written is meaningless hocum. Sci-Gen - https://pdos.csail.mit.edu/archive/scigen/ https://pdos.csail.mit.edu/archive/scigen/ Reporting on withdrawals of papers - https://www.researchgate.net/publication/278619529_Detection_of_computer_generated_papers_in_scientific_literature https://www.researchgate.net/publication/278619529_Detection...

29athrowaway 6y ago
Dr. Herbert Schlangemann (fake alias for Sci-gen) not only got papers accepted in journals, it was invited to participate as session chair at conference co-sponsored by IEEE. https://en.wikipedia.org/wiki/SCIgen#Schlangemann https://en.wikipedia.org/wiki/SCIgen#Schlangemann

pasabagi 6y ago

Kind of annoying 'hoax', though. Obviously you can publish garbage in fringe journals if you leverage pre-existing prestige and position like Sokal did. Doesn't really say anything about the social sciences. You can also publish a lot of nonsense in certain chinese journals that optimize for quantity in quality, in whatever field you want.

yaj54 6y ago

The thing that kills me is that soon enough, a "fake" paper written by GPT-3 will get published in an academic journal because it has actually contributed a new insight. It's easy to consider text generation models as "just mimicking grammar". But isn't grammar also just a model of human cognition? Is GPT modeling grammar or is it modeling human cognition? Since GPT can ingest radically more text (aka ideas) won't it soon be able to generate texts (aka ideas) that are a more accurate collation of current knowledge than any individual human could generate? -- [Was this comment written by GPT-3?]

Udik 6y ago
My impression is that these models are already doing far more than what the language production machinery in our brain does. We are able to produce language according to grammar and semantics, but we also have independent mental representations to guide the generation of language and to provide context. I don't really understand why we're trying so hard to build models that can generate coherent texts based on having predigested only other texts, without any other experience of reality. Their capabilities appear already superhuman in their ability to imitate styles and patterns of any kind (including code generation, images, etc.). It feels like we're overshooting our target by trying to solve an unsolvable problem, that of deriving the semantics of reality from pure text, without any other type of input.
lostmsu 6y ago
It was not - there's a point :-D I am impressed though nobody dared to guess in 2 weeks.