7 ms·

I swear this kind of stuff is only going to get worse as people come to use and rely on ChatGPT and its ilk more broadly. Dystopia? Idiocracy? I don't know, bu

I swear this kind of stuff is only going to get worse as people come to use and rely on ChatGPT and its ilk more broadly. Dystopia? Idiocracy? I don't know, but I don't like it.

bobmaxup 4y ago
Another problem to be solved with more computation and human backed reinforcement learning, surely.
- irrational 4y ago
  How? As can be seen from these Citogenesis Incidents, humans cannot even tell when other humans are making up stuff that sounds like it could be real. How will ChatGPT, et al do it?
echelon 4y ago
It'll probably be a little bit like life before the web. You'd hear something and have no immediate way to verify the veracity of the claim. It's one of the reasons teachers pushed students to go to the library and use encyclopedias for citations when writing research papers. Our species has obviously managed to make it pretty far without facts for the longest time. But we've comfortably lived with easily verified facts for 20-30 years and are now faced with a return to uncertainty. If I had to guess, we'll see stricter controls on institutions such as Wikipedia that rely on credentialism and frequent auditing as a means to counter the new at-volume information creation capacity. But I don't really have the faintest idea of how this will turn out yet. It's wild to think about how much things are changing.
- tjohns 4y ago
  My teachers were always clear that encyclopedias were not to be used as a primary source either. They're not even a secondary source. Encyclopedias are tertiary sources. They're better than Wikipedia... but only barely. In the end you use Wikipedia and an encyclopedia the same way: to get a broad understanding of a topic as a mental framework, then look at the article's citations as a starting point to find actual, citable primary sources. (Plus the rest of the library's catalog/databases.)
  - weaksauce 4y ago
    exactly. information literacy starts with evaluating the sources. I have had numerous chats over the last few years where it's evident that people do not do due dillagence in their information gathering. it seems that either people aren't being taught this anymore or that they have given in to sloppy thinking.
chess_buster 4y ago
Imagine a world where children have grown up, relying on ChatGPT for each and every question.
- throw124 4y ago
  Imagine that in five years from now, ChatGPT or one of its competitors will reach 98% factual accuracy in responses. Would you not like to rely on it for answering your questions?
  - VonGallifrey 4y ago
    Saying this in a discussion about Citogenesis is funny to me. How would you even determine "factual accuracy"? Just look at the list. There are many instances where "reliable sources" repeated false information which was then used to "prove" that the information is reliable. As far as I am concerned AI responses will never be reliable without verification. Same as any human responses, but there you can at least verify credentials.
  - paisawalla 4y ago
    Scroll down TFA to the section called "terms that became real". When trolls or adversaries can use citogenesis to boostrap facts into the mainstream from a cold start, what does "98% factual accuracy" mean? At some point, you'll have to include the "formerly known as BS" facts.
  - lm28469 4y ago
    Outside of maths and physics there is no such thing as factual truths
    - irrational 4y ago
      Isn't there? I didn't attend MIT. That is a factual truth.
      - derekp7 4y ago
deleted 4y ago
[deleted]
DoreenMichele 4y ago
"This stuff" has been around for ages. Forty years ago in my teens, I read a book with a very memorable anecdote about locals insisting canabalism was "a local custom" and defending their right to practice it. The British official in charge replied "It's our custom to shoot cannibals." When I was homeless, I asked around on the internet for a source for who said that. There is a real incident where a British official said something like "We shoot people who do that" but it wasn't about cannibalism. But the toxic classist forum where I asked initially replied to me with basically "You're just a stupid homeless person misremembering that." No, I read it in my teens when I had a near photographic memory and was one of the top students in my high school class and everyone respected me as one of the smart people, long before the world decided I was some loser making things up. I'm quite clear the anecdote in the book was about cannibalism. There is a real historical incident similar to it, but the book got the details wrong. I have also read some crazy accounts of how Einstein's Theory was proved because the solar eclipse bent light so much, you could see a star that our Sun should have been obscuring. Humans tend to believe things we read. If it's in writing, it has some kind of authority in our minds. This is often not the case and we need to get better about recognizing that a lot of "writing" on the internet is just modern chit chat and not reliable.
- rvba 4y ago
  How did the people on the board know you were homeless and why do you keep randomly keep mentioning it?
  - DoreenMichele 4y ago
    Do I have some social obligation to hide the fact that I was homeless at one time?
    - Brian_K_White 4y ago
      It never occurred to me that there was a rule about public displays of residence like for public displays of affection, but I guess there is. (inappropriate and maybe even criminal in the presense of children, unless you're straight & monogamous, in that case it's fine)

I swear this kind of stuff is only going to get worse as people come to use and rely on ChatGPT and its ilk more broadly. Dystopia? Idiocracy? I don't know, but I don't like it.

bobmaxup 4y ago

Another problem to be solved with more computation and human backed reinforcement learning, surely.

irrational 4y ago
How? As can be seen from these Citogenesis Incidents, humans cannot even tell when other humans are making up stuff that sounds like it could be real. How will ChatGPT, et al do it?

echelon 4y ago

It'll probably be a little bit like life before the web. You'd hear something and have no immediate way to verify the veracity of the claim. It's one of the reasons teachers pushed students to go to the library and use encyclopedias for citations when writing research papers. Our species has obviously managed to make it pretty far without facts for the longest time. But we've comfortably lived with easily verified facts for 20-30 years and are now faced with a return to uncertainty. If I had to guess, we'll see stricter controls on institutions such as Wikipedia that rely on credentialism and frequent auditing as a means to counter the new at-volume information creation capacity. But I don't really have the faintest idea of how this will turn out yet. It's wild to think about how much things are changing.

tjohns 4y ago
My teachers were always clear that encyclopedias were not to be used as a primary source either. They're not even a secondary source. Encyclopedias are tertiary sources. They're better than Wikipedia... but only barely. In the end you use Wikipedia and an encyclopedia the same way: to get a broad understanding of a topic as a mental framework, then look at the article's citations as a starting point to find actual, citable primary sources. (Plus the rest of the library's catalog/databases.)
- weaksauce 4y ago
  exactly. information literacy starts with evaluating the sources. I have had numerous chats over the last few years where it's evident that people do not do due dillagence in their information gathering. it seems that either people aren't being taught this anymore or that they have given in to sloppy thinking.

chess_buster 4y ago

Imagine a world where children have grown up, relying on ChatGPT for each and every question.

throw124 4y ago
Imagine that in five years from now, ChatGPT or one of its competitors will reach 98% factual accuracy in responses. Would you not like to rely on it for answering your questions?
- VonGallifrey 4y ago
  Saying this in a discussion about Citogenesis is funny to me. How would you even determine "factual accuracy"? Just look at the list. There are many instances where "reliable sources" repeated false information which was then used to "prove" that the information is reliable. As far as I am concerned AI responses will never be reliable without verification. Same as any human responses, but there you can at least verify credentials.
- paisawalla 4y ago
  Scroll down TFA to the section called "terms that became real". When trolls or adversaries can use citogenesis to boostrap facts into the mainstream from a cold start, what does "98% factual accuracy" mean? At some point, you'll have to include the "formerly known as BS" facts.
- lm28469 4y ago
  Outside of maths and physics there is no such thing as factual truths
  - irrational 4y ago
    Isn't there? I didn't attend MIT. That is a factual truth.
    - derekp7 4y ago

deleted 4y ago

[deleted]

DoreenMichele 4y ago

"This stuff" has been around for ages. Forty years ago in my teens, I read a book with a very memorable anecdote about locals insisting canabalism was "a local custom" and defending their right to practice it. The British official in charge replied "It's our custom to shoot cannibals." When I was homeless, I asked around on the internet for a source for who said that. There is a real incident where a British official said something like "We shoot people who do that" but it wasn't about cannibalism. But the toxic classist forum where I asked initially replied to me with basically "You're just a stupid homeless person misremembering that." No, I read it in my teens when I had a near photographic memory and was one of the top students in my high school class and everyone respected me as one of the smart people, long before the world decided I was some loser making things up. I'm quite clear the anecdote in the book was about cannibalism. There is a real historical incident similar to it, but the book got the details wrong. I have also read some crazy accounts of how Einstein's Theory was proved because the solar eclipse bent light so much, you could see a star that our Sun should have been obscuring. Humans tend to believe things we read. If it's in writing, it has some kind of authority in our minds. This is often not the case and we need to get better about recognizing that a lot of "writing" on the internet is just modern chit chat and not reliable.

rvba 4y ago
How did the people on the board know you were homeless and why do you keep randomly keep mentioning it?
- DoreenMichele 4y ago
  Do I have some social obligation to hide the fact that I was homeless at one time?
  - Brian_K_White 4y ago
    It never occurred to me that there was a rule about public displays of residence like for public displays of affection, but I guess there is. (inappropriate and maybe even criminal in the presense of children, unless you're straight & monogamous, in that case it's fine)