10 ms·

List of Citogenesis Incidents

169 points by Thevet 4y ago

sva_ 4y ago
Somehow reminds me of large language models. If they will be trained on data after the release of, say, GPT-3, they'll probably be trained on outputs of that model.
- throwbadubadu 4y ago
  Yes, first thing that comes to mind: How much worse will this become with LLMs in the loop - almost assuming that this page was even submitted for that thought?
- r3trohack3r 4y ago
  Somehow I don't think this is going to be a problem. I can't exactly articulate why, but I'm going to try. The success of an LLM is quite subjective. We have metrics that try to quantitatively measure the performance of an LLM, but the "real" test are the users that the LLM does work for. Those users are ultimately human, even if there are layers and layers of LLMs collaborating under a human interface. I think what ultimately matters is that the output is considered high quality by the end user. I don't think that it actually matters if an input is AI generated or human generated when training a model, as long as the LLM continues producing high quality results. I think implicit in your argument is that the _quality_ of the _training set_ is going to deteriorate due to LLM generated content. But: 1) I don't know how much quality of the input actually impacts the outcome. Almost certainly an entire corpus of noise isn't going to generate signal when passed through an LLM, but what an acceptable signal/noise ratio is seems to be an unanswered question. 2) AI generated content doesn't necessarily mean it is low quality content. In fact, if we find a high quality training set yields substantially better AI, I'd rather have a training set of 100% AI generated content that is human reviewed to be high quality vs. one that is 100% human generated content but unfiltered for quality. I don't necessarily think this feedback loop, of LLM outputs feeding LLM inputs, is necessarily the problem people say it is. But might be wrong!
  - aezart 4y ago
    LLM output cannot be higher quality than the input (prompt + training data). The best possible outcome for an LLM is that the output is a correct continuation of the prompt. The output will usually be a less-than-perfect continuation. With small models, at least, you can watch LLM output degrade in real time as more text is generated, because the ratio of prompt to output in the context gets smaller with each new token. So the LLM is trying to imitate itself, more than it is trying to imitate the prompt. Bigger models can't fix this problem, they can just slow down the rate of degradation. It's bad enough when the model is stuck trying to imitate its output in the current context, but it'll be much worse if it's actually fed back in as training data. In that scenario, the bad data poisons all future output from the model, not just the current context.
    - 93po 4y ago
      Unless the LLM actually evaluates sources and assigns weights to sources.
    - Kye 4y ago
      This is interesting because it's essentially how human bullshitters work. The more they know, the longer they can convince you they know more than they do. https://xkcd.com/451/ https://xkcd.com/451/ Imposter sophistication levels might be the way we rank these in the future.
    - roflyear 4y ago
      But haven't you used these models? They are baby AGI! They'll grow up!! /s
- 93po 4y ago
  It's a major area of focus to improve the hallucination (or whatever the technically correct term is) of the model. I would bet we're pretty close to GPT actually evaluating sources for information and making judgements in how to weight those sources. I suspect this is going to upset a lot of people, and especially those in power.
  - deleted 4y ago
    [deleted]
greenyoda 4y ago
Here's a fictitious citation that commonly appears on HN - "Dunning-Kruger effect": > The expression "Dunning–Kruger effect" was created on Wikipedia in May 2006, in this edit.[1] The article had been created in July 2005 as Dunning-Kruger Syndrome. Neither of these terms appeared at that time in scientific literature; the "syndrome" name was created to summarise the findings of one 1999 paper by David Dunning and Justin Kruger. The change to "effect" was not prompted by any sources, but by a concern that "syndrome" would falsely imply a medical condition. By the time the article name was criticised as original research in 2008, Google Scholar was showing a number of academic sources describing the Dunning–Kruger effect using explanations similar to the Wikipedia article.[2] [1] https://en.wikipedia.org/w/index.php?diff=55273744&diffmode=source https://en.wikipedia.org/w/index.php?diff=55273744&diffmode=... [2] https://en.wikipedia.org/wiki/Wikipedia:List_of_citogenesis_incidents#Terms_that_became_real https://en.wikipedia.org/wiki/Wikipedia:List_of_citogenesis_...
- The28thDuck 4y ago
  Does that mean the Dunning-Krueger effect has no basis in truth?
  - JohnBerea 4y ago
    I'm from the internet and I can assure you it has no basis in truth.
  - maxbond 4y ago
    ETA: see response Probably not no basis, Dunning and Krueger really did so research & found [retracted] a negative correlation between self-rated ability and performance on an aptitude test afaik [/retracted]. But it's often overgeneralized or taken to be some kind of law rather than an observation.
    - dragonwriter 4y ago
      > Dunning and Krueger really did so research & find a negative correlation between self-rated ability and performance on an aptitude test afaik No, they didn't. They found a positive linear relation with between actual and self-assessed relative performance, with the intersection point at around the 70th percentile. (That is, people on average report themselves closer to the 70th percentile than they are, those below erring higher and those above erring lower.) The (self-rated rank) - (actual rank) difference goes up as actual rank goes down, but that's not self-rated ability going up with reduced ability.
renewiltord 4y ago
A friend of mine intercepted one that was actually shared in other places: https://en.wikipedia.org/w/index.php?title=Said_the_actress_to_the_bishop&diff=prev&oldid=1102363330&diffmode=source https://en.wikipedia.org/w/index.php?title=Said_the_actress_...
petercooper 4y ago
I just asked ChatGPT who made the first cardboard box, and it too believes the first story on this list: "The first cardboard box was invented by Sir Malcolm Thornhill in England in 1817. He created a machine that could make sheets of paper and then fold them into boxes."
- TheRealPomax 4y ago
  No, it doesn't. It doesn't believe anything, it's just generating a story for you that sounds credible enough for you to go "yes, this is what an answer would look like". That's its job. That's its only job. Literally everything it says is fabricated, and if it happens to be the truth, that's a coincidence.
  - petercooper 4y ago
    Literally everything it says is fabricated In what sense do you use the word "fabricated"? In the sense that it invented a falsehood with an intent to deceive, or in that it says things based upon prior exposure?
    - spartanatreyu 4y ago
      Fabricated as in manufactured, in the same way that a hallucination fabricates/manufactures information. There's no intent to deceive because hallucinations have no capacity for intent.
  - nitwit005 4y ago
    It effectively "believes" some things, as it will consistently emit certain statements in response to certain types of queries. It considers that information part of a good response. There is information stored in its model. That information might not be correct.
    - Brian_K_White 4y ago
      An mp3 player repeating something does not believe it. There is no way even at any level of abstraction and squinting just right to use a term like that for what chatgpt is or does. It's fancy auto-complete. Literally matching and mashing up patterns against other writings by probability. That's it. Auto-complete is neither understanding nor believing.
irrational 4y ago
I swear this kind of stuff is only going to get worse as people come to use and rely on ChatGPT and its ilk more broadly. Dystopia? Idiocracy? I don't know, but I don't like it.
- bobmaxup 4y ago
  Another problem to be solved with more computation and human backed reinforcement learning, surely.
  - irrational 4y ago
    How? As can be seen from these Citogenesis Incidents, humans cannot even tell when other humans are making up stuff that sounds like it could be real. How will ChatGPT, et al do it?
- echelon 4y ago
  It'll probably be a little bit like life before the web. You'd hear something and have no immediate way to verify the veracity of the claim. It's one of the reasons teachers pushed students to go to the library and use encyclopedias for citations when writing research papers. Our species has obviously managed to make it pretty far without facts for the longest time. But we've comfortably lived with easily verified facts for 20-30 years and are now faced with a return to uncertainty. If I had to guess, we'll see stricter controls on institutions such as Wikipedia that rely on credentialism and frequent auditing as a means to counter the new at-volume information creation capacity. But I don't really have the faintest idea of how this will turn out yet. It's wild to think about how much things are changing.
  - tjohns 4y ago
    My teachers were always clear that encyclopedias were not to be used as a primary source either. They're not even a secondary source. Encyclopedias are tertiary sources. They're better than Wikipedia... but only barely. In the end you use Wikipedia and an encyclopedia the same way: to get a broad understanding of a topic as a mental framework, then look at the article's citations as a starting point to find actual, citable primary sources. (Plus the rest of the library's catalog/databases.)
malkia 4y ago
Reminds me of https://www.reddit.com/r/Jokes/comments/2wpf2h/indian_joke_about_weather/ https://www.reddit.com/r/Jokes/comments/2wpf2h/indian_joke_a... (sed /indian/native american/g)
shaftoe444 4y ago
Really hoped that the fact this came from xkcd was itself an example of citogenesis.
1970-01-01 4y ago
Deadspin saw one of these and made with a step-by-step summary: https://deadspin.com/how-espn-manufactures-a-story-colin-kaepernick-edition-1185400028 https://deadspin.com/how-espn-manufactures-a-story-colin-kae...
dszoboszlay 4y ago
> arbitrary addition to Coati, "also known as....the Brazilian aardvark" But via citogenesis, the coati really became also known as the Brazilian aardvark. So the original claim is true, and this wasn't really citogenesis after all. More like self fulfilling prophecy.
- thaumasiotes 4y ago
  In linguistics a sentence that causes itself to be true is called performative. (The standard example being "we are at war", spoken by someone with the authority to declare war.) That doesn't really fit here, but it's a similar idea.
realworldperson 4y ago
[dead]
93po 4y ago
The 85% fatality rate for the water speed record had me go down a rabbit hole. The record hasn't been broken since 1978 (~315mph) and someone on reddit said it was because they stopped tracking the record due to so many deaths. I can't find any information online to corroborate this though.
Berniek 4y ago
The references in the comments suggest ChatGPT as providing this effect. But that is (or should be) unlikely, the "training" or moderation (tweaking?) should actually solve this problem. It should be relatively easy to separate it's own generation from sources. BUT where it will happen is when multiple instances of these language models compete with each other. ChatGPT quoting Bing or Bard output probably can't be reliably countered with internal training of ChatGTP, and the same goes for Bing & Bard and all the other myriad manifestations of these data mining techniques. (Unless they merge them togther?)
- uoaei 4y ago
  > BUT where it will happen is when multiple instances of these language models compete with each other. That's what everyone else is saying already. Not sure what exactly you are arguing against.
  - Berniek 4y ago
    Sorry a bit late replying-been away. It is not the competition that is bad, it that anything produced by them cannot be tweaked and so become "circular" sources. There will be no way to test for "truth", at least on an individual bot the training data can be tweaked to not use its own production as source. The competition will make the validity or "truth" of most data questionable. I guess it should be possible for an individual LMM to be trained for "truth" (reality?) but it becomes almost impossible for a LMM to discern truth when the sources it is analyzing are of generated by another LMM
nitwit005 4y ago
> Terms that became real I suppose that was inevitable.
mjd 4y ago
In 2016 I discovered that a mountain on the planet Ceres had officially been named "Ysolo Mons" after the Albanian festival that marks the start of the annual eggplant harvest. There is no such festival. An anonymous Wikipedia editor had made it up and inserted it into Wikipedia's list of harvest festivals in 2012. Someone at NASA used the Wikipedia list for naming features on Ceres. (Ceres was the Roman goddess of agriculture.) I wrote to the US geological survey to point this out. They changed the name of the mountain. Full story on my blog: https://blog.plover.com/wikipedia/ysolo.html https://blog.plover.com/wikipedia/ysolo.html
humanpotato 4y ago
Many years ago I discovered and wrote the bullet in this article about the Zimbabwean hyperinflation: "It is well known that Zimbabwe experienced severe hyperinflation in 2008..." Very interesting article as a whole, of course.

169 points by Thevet 4y ago

sva_ 4y ago

Somehow reminds me of large language models. If they will be trained on data after the release of, say, GPT-3, they'll probably be trained on outputs of that model.

throwbadubadu 4y ago
Yes, first thing that comes to mind: How much worse will this become with LLMs in the loop - almost assuming that this page was even submitted for that thought?
r3trohack3r 4y ago
Somehow I don't think this is going to be a problem. I can't exactly articulate why, but I'm going to try. The success of an LLM is quite subjective. We have metrics that try to quantitatively measure the performance of an LLM, but the "real" test are the users that the LLM does work for. Those users are ultimately human, even if there are layers and layers of LLMs collaborating under a human interface. I think what ultimately matters is that the output is considered high quality by the end user. I don't think that it actually matters if an input is AI generated or human generated when training a model, as long as the LLM continues producing high quality results. I think implicit in your argument is that the _quality_ of the _training set_ is going to deteriorate due to LLM generated content. But: 1) I don't know how much quality of the input actually impacts the outcome. Almost certainly an entire corpus of noise isn't going to generate signal when passed through an LLM, but what an acceptable signal/noise ratio is seems to be an unanswered question. 2) AI generated content doesn't necessarily mean it is low quality content. In fact, if we find a high quality training set yields substantially better AI, I'd rather have a training set of 100% AI generated content that is human reviewed to be high quality vs. one that is 100% human generated content but unfiltered for quality. I don't necessarily think this feedback loop, of LLM outputs feeding LLM inputs, is necessarily the problem people say it is. But might be wrong!
- aezart 4y ago
  LLM output cannot be higher quality than the input (prompt + training data). The best possible outcome for an LLM is that the output is a correct continuation of the prompt. The output will usually be a less-than-perfect continuation. With small models, at least, you can watch LLM output degrade in real time as more text is generated, because the ratio of prompt to output in the context gets smaller with each new token. So the LLM is trying to imitate itself, more than it is trying to imitate the prompt. Bigger models can't fix this problem, they can just slow down the rate of degradation. It's bad enough when the model is stuck trying to imitate its output in the current context, but it'll be much worse if it's actually fed back in as training data. In that scenario, the bad data poisons all future output from the model, not just the current context.
  - 93po 4y ago
    Unless the LLM actually evaluates sources and assigns weights to sources.
  - Kye 4y ago
    This is interesting because it's essentially how human bullshitters work. The more they know, the longer they can convince you they know more than they do. https://xkcd.com/451/ https://xkcd.com/451/ Imposter sophistication levels might be the way we rank these in the future.
  - roflyear 4y ago
    But haven't you used these models? They are baby AGI! They'll grow up!! /s
93po 4y ago
It's a major area of focus to improve the hallucination (or whatever the technically correct term is) of the model. I would bet we're pretty close to GPT actually evaluating sources for information and making judgements in how to weight those sources. I suspect this is going to upset a lot of people, and especially those in power.
- deleted 4y ago
  [deleted]

greenyoda 4y ago

Here's a fictitious citation that commonly appears on HN - "Dunning-Kruger effect": > The expression "Dunning–Kruger effect" was created on Wikipedia in May 2006, in this edit.[1] The article had been created in July 2005 as Dunning-Kruger Syndrome. Neither of these terms appeared at that time in scientific literature; the "syndrome" name was created to summarise the findings of one 1999 paper by David Dunning and Justin Kruger. The change to "effect" was not prompted by any sources, but by a concern that "syndrome" would falsely imply a medical condition. By the time the article name was criticised as original research in 2008, Google Scholar was showing a number of academic sources describing the Dunning–Kruger effect using explanations similar to the Wikipedia article.[2] [1] https://en.wikipedia.org/w/index.php?diff=55273744&diffmode=source https://en.wikipedia.org/w/index.php?diff=55273744&diffmode=... [2] https://en.wikipedia.org/wiki/Wikipedia:List_of_citogenesis_incidents#Terms_that_became_real https://en.wikipedia.org/wiki/Wikipedia:List_of_citogenesis_...

The28thDuck 4y ago
Does that mean the Dunning-Krueger effect has no basis in truth?
- JohnBerea 4y ago
  I'm from the internet and I can assure you it has no basis in truth.
- maxbond 4y ago
  ETA: see response Probably not no basis, Dunning and Krueger really did so research & found [retracted] a negative correlation between self-rated ability and performance on an aptitude test afaik [/retracted]. But it's often overgeneralized or taken to be some kind of law rather than an observation.
  - dragonwriter 4y ago
    > Dunning and Krueger really did so research & find a negative correlation between self-rated ability and performance on an aptitude test afaik No, they didn't. They found a positive linear relation with between actual and self-assessed relative performance, with the intersection point at around the 70th percentile. (That is, people on average report themselves closer to the 70th percentile than they are, those below erring higher and those above erring lower.) The (self-rated rank) - (actual rank) difference goes up as actual rank goes down, but that's not self-rated ability going up with reduced ability.

renewiltord 4y ago

A friend of mine intercepted one that was actually shared in other places: https://en.wikipedia.org/w/index.php?title=Said_the_actress_to_the_bishop&diff=prev&oldid=1102363330&diffmode=source https://en.wikipedia.org/w/index.php?title=Said_the_actress_...

petercooper 4y ago

I just asked ChatGPT who made the first cardboard box, and it too believes the first story on this list: "The first cardboard box was invented by Sir Malcolm Thornhill in England in 1817. He created a machine that could make sheets of paper and then fold them into boxes."

TheRealPomax 4y ago
No, it doesn't. It doesn't believe anything, it's just generating a story for you that sounds credible enough for you to go "yes, this is what an answer would look like". That's its job. That's its only job. Literally everything it says is fabricated, and if it happens to be the truth, that's a coincidence.
- petercooper 4y ago
  Literally everything it says is fabricated In what sense do you use the word "fabricated"? In the sense that it invented a falsehood with an intent to deceive, or in that it says things based upon prior exposure?
  - spartanatreyu 4y ago
    Fabricated as in manufactured, in the same way that a hallucination fabricates/manufactures information. There's no intent to deceive because hallucinations have no capacity for intent.
- nitwit005 4y ago
  It effectively "believes" some things, as it will consistently emit certain statements in response to certain types of queries. It considers that information part of a good response. There is information stored in its model. That information might not be correct.
  - Brian_K_White 4y ago
    An mp3 player repeating something does not believe it. There is no way even at any level of abstraction and squinting just right to use a term like that for what chatgpt is or does. It's fancy auto-complete. Literally matching and mashing up patterns against other writings by probability. That's it. Auto-complete is neither understanding nor believing.

irrational 4y ago

I swear this kind of stuff is only going to get worse as people come to use and rely on ChatGPT and its ilk more broadly. Dystopia? Idiocracy? I don't know, but I don't like it.

bobmaxup 4y ago
Another problem to be solved with more computation and human backed reinforcement learning, surely.
- irrational 4y ago
  How? As can be seen from these Citogenesis Incidents, humans cannot even tell when other humans are making up stuff that sounds like it could be real. How will ChatGPT, et al do it?
echelon 4y ago
It'll probably be a little bit like life before the web. You'd hear something and have no immediate way to verify the veracity of the claim. It's one of the reasons teachers pushed students to go to the library and use encyclopedias for citations when writing research papers. Our species has obviously managed to make it pretty far without facts for the longest time. But we've comfortably lived with easily verified facts for 20-30 years and are now faced with a return to uncertainty. If I had to guess, we'll see stricter controls on institutions such as Wikipedia that rely on credentialism and frequent auditing as a means to counter the new at-volume information creation capacity. But I don't really have the faintest idea of how this will turn out yet. It's wild to think about how much things are changing.
- tjohns 4y ago
  My teachers were always clear that encyclopedias were not to be used as a primary source either. They're not even a secondary source. Encyclopedias are tertiary sources. They're better than Wikipedia... but only barely. In the end you use Wikipedia and an encyclopedia the same way: to get a broad understanding of a topic as a mental framework, then look at the article's citations as a starting point to find actual, citable primary sources. (Plus the rest of the library's catalog/databases.)

malkia 4y ago

Reminds me of https://www.reddit.com/r/Jokes/comments/2wpf2h/indian_joke_about_weather/ https://www.reddit.com/r/Jokes/comments/2wpf2h/indian_joke_a... (sed /indian/native american/g)

shaftoe444 4y ago

Really hoped that the fact this came from xkcd was itself an example of citogenesis.

1970-01-01 4y ago

Deadspin saw one of these and made with a step-by-step summary: https://deadspin.com/how-espn-manufactures-a-story-colin-kaepernick-edition-1185400028 https://deadspin.com/how-espn-manufactures-a-story-colin-kae...

dszoboszlay 4y ago

> arbitrary addition to Coati, "also known as....the Brazilian aardvark" But via citogenesis, the coati really became also known as the Brazilian aardvark. So the original claim is true, and this wasn't really citogenesis after all. More like self fulfilling prophecy.

thaumasiotes 4y ago
In linguistics a sentence that causes itself to be true is called performative. (The standard example being "we are at war", spoken by someone with the authority to declare war.) That doesn't really fit here, but it's a similar idea.

realworldperson 4y ago

[dead]

93po 4y ago

The 85% fatality rate for the water speed record had me go down a rabbit hole. The record hasn't been broken since 1978 (~315mph) and someone on reddit said it was because they stopped tracking the record due to so many deaths. I can't find any information online to corroborate this though.

Berniek 4y ago

The references in the comments suggest ChatGPT as providing this effect. But that is (or should be) unlikely, the "training" or moderation (tweaking?) should actually solve this problem. It should be relatively easy to separate it's own generation from sources. BUT where it will happen is when multiple instances of these language models compete with each other. ChatGPT quoting Bing or Bard output probably can't be reliably countered with internal training of ChatGTP, and the same goes for Bing & Bard and all the other myriad manifestations of these data mining techniques. (Unless they merge them togther?)

uoaei 4y ago
> BUT where it will happen is when multiple instances of these language models compete with each other. That's what everyone else is saying already. Not sure what exactly you are arguing against.
- Berniek 4y ago
  Sorry a bit late replying-been away. It is not the competition that is bad, it that anything produced by them cannot be tweaked and so become "circular" sources. There will be no way to test for "truth", at least on an individual bot the training data can be tweaked to not use its own production as source. The competition will make the validity or "truth" of most data questionable. I guess it should be possible for an individual LMM to be trained for "truth" (reality?) but it becomes almost impossible for a LMM to discern truth when the sources it is analyzing are of generated by another LMM

nitwit005 4y ago

> Terms that became real I suppose that was inevitable.

mjd 4y ago

In 2016 I discovered that a mountain on the planet Ceres had officially been named "Ysolo Mons" after the Albanian festival that marks the start of the annual eggplant harvest. There is no such festival. An anonymous Wikipedia editor had made it up and inserted it into Wikipedia's list of harvest festivals in 2012. Someone at NASA used the Wikipedia list for naming features on Ceres. (Ceres was the Roman goddess of agriculture.) I wrote to the US geological survey to point this out. They changed the name of the mountain. Full story on my blog: https://blog.plover.com/wikipedia/ysolo.html https://blog.plover.com/wikipedia/ysolo.html

humanpotato 4y ago

Many years ago I discovered and wrote the bullet in this article about the Zimbabwean hyperinflation: "It is well known that Zimbabwe experienced severe hyperinflation in 2008..." Very interesting article as a whole, of course.