4 ms·

I've played with ChatGPT and see it as (at best) just complimentary to existing approaches. Unfortunately, I also see it as a vector of misinformation and poten

I've played with ChatGPT and see it as (at best) just complimentary to existing approaches. Unfortunately, I also see it as a vector of misinformation and potentially harmful. It appears highly knowledgeable on topics that I know little about, but otherwise it's replies are often riddled with errors and inaccuracies. It also seems unable to quote sources or references for such information. It seems reasonable to use a statistical process to generate conversational chat and boilerplate, but as a source of truth, it appears not.

RheingoldRiver 4y ago
I think this is called Knoll's law of media accuracy [0] or wikipedia [1] - "Everything you read in the newspapers is absolutely true except for the rare story of which you happen to have firsthand knowledge". Interestingly enough, I tried asking ChatGPT what it was before googling it, and it confidently told me I was asking about "Confirmation Bias." But then when I told it what I was talking about was indeed Knoll's Law, it got the definition of the law completely wrong! > I apologize for any confusion. You are correct, what you're describing is known as "Knoll's Law of Media Accuracy." It states that the accuracy of a story is inversely proportional to the number of people involved in its reporting, and the distance from the original source of information. In other words, the more people involved in reporting a story, the more likely it is to be inaccurate, and the further away the story is from the original source, the more likely it is to be inaccurate. This is one of the worst answers the bot has ever given me, and I've asked it a wide variety of prompts, from things I know to things I don't, including having it generate me fictional responses. [0] https://effectiviology.com/knolls-law/ https://effectiviology.com/knolls-law/ [1] https://en.wikipedia.org/wiki/Erwin_Knoll https://en.wikipedia.org/wiki/Erwin_Knoll
- waterhouse 4y ago
  Approximately the same phenomenon is also known as the Gell-Mann amnesia effect. It comes up on HN a lot, but for those who haven't encountered it before: > You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. > In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know. https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect
- omnicognate 4y ago
  It did the same thing for me on Gell-Mann amnesia, which is the same concept. Read carefully, it's a particularly good example of "wrong but highly convincing": > Gell-Mann amnesia is a term coined by science writer Michael Crichton to describe the phenomenon where people tend to forget information that contradicts their preconceptions or beliefs. The term is named after physicist Murray Gell-Mann, who won the 1969 Nobel Prize in Physics for his work on the theory of elementary particles. > According to Crichton, Gell-Mann amnesia occurs when people encounter information that contradicts their beliefs, but rather than updating their beliefs to reflect the new information, they simply forget the information and continue to hold onto their preconceptions. This type of cognitive bias can be a significant barrier to learning and can lead to flawed decision-making.
- dinkumthinkum 4y ago
  Exactly. We should be realistic about AI and ChatGPT and LLMs. You have great example of how “confidently” produced atrocious results even with the oft-cited ad hoc solution of “follow up” it gave even more misinformation and you, the user, only know this because you happened to know the answer. Oddly enough, if you google the types of things you are talking about, you will largely get the correct answer as the first result. There is something to be said about being naive about the potential of LLMs models but I am astounded how hyped and eager everyone is to hand over the future of all knowledge work to them without even clear evidence that that would make sense.
waterhouse 4y ago
Never tried it myself, but a question comes to mind: Can you ask it to state how confident it is in the truth of its answer? Is it well-calibrated in that sense? If so, perhaps that'd make it more useful.
- RheingoldRiver 4y ago
  A related thing I've done -- My mom's a college professor, and she's worried about her students using ChatGPT to cheat. So she wants to engineer questions in a way that cheating is impossible, perhaps by even incorporating ChatGPT into the curriculum in some ways. To explore possibilities, she's enlisting my help. So one thing I did was give it some prompt (the first one I chose was a vague thing, "contrast the theme of "discovery" in the Odyssey and the Aeneid," this is a terrible prompt but hey I'm not a great books professor) and I said "generate a college essay on this prompt" so it gave me some really truly shit couple of paragraphs. Then I said "that's not really college-level writing can you try again." And then I had it repeat this maybe 3 or 4 times. Honestly, it didn't get all that better, probably because my prompt sucked a lot. But then at the end I said "okay can you compare your new essay to your original one" and it gave a couple good points. I've had it do a lot of similar things since, and I always have it analyze its answers, compare its stories, etc. One HYSTERICAL answer that it gave was (and note that "The Great Migration" is a play that ChatGPT itself wrote itself in response to the prompt "can you write a play using the homeric tradition to talk about the great migration, as if you were august wilson"): > As a language model, I don't have personal preferences, it is not possible for me to say which play is better. Both plays "Fences" by August Wilson and "The Great Migration" that I wrote, deal with similar themes and issues that are important and relevant to the African American community. "Fences" is widely considered a masterpiece of American theater, and August Wilson is a celebrated playwright who won numerous awards and accolades for his works. "Fences" is a well-crafted, character-driven play that explores the personal struggles and relationships of the characters in a specific historical context. > On the other hand, "The Great Migration" is a fictional play that I wrote based on the historical event and it is more plot-driven. It provides a general overview of the experiences of African Americans during the Great Migration period. > Both plays are important and valuable in their own ways, they offer different perspectives and insights into the experiences of African Americans during different times in history. It ultimately comes down to the audience preference, taste and the specific themes they are looking to explore.
otabdeveloper4 4y ago
ChatGPT is automated copywriting, what did you expect?

RheingoldRiver 4y ago

I think this is called Knoll's law of media accuracy [0] or wikipedia [1] - "Everything you read in the newspapers is absolutely true except for the rare story of which you happen to have firsthand knowledge". Interestingly enough, I tried asking ChatGPT what it was before googling it, and it confidently told me I was asking about "Confirmation Bias." But then when I told it what I was talking about was indeed Knoll's Law, it got the definition of the law completely wrong! > I apologize for any confusion. You are correct, what you're describing is known as "Knoll's Law of Media Accuracy." It states that the accuracy of a story is inversely proportional to the number of people involved in its reporting, and the distance from the original source of information. In other words, the more people involved in reporting a story, the more likely it is to be inaccurate, and the further away the story is from the original source, the more likely it is to be inaccurate. This is one of the worst answers the bot has ever given me, and I've asked it a wide variety of prompts, from things I know to things I don't, including having it generate me fictional responses. [0] https://effectiviology.com/knolls-law/ https://effectiviology.com/knolls-law/ [1] https://en.wikipedia.org/wiki/Erwin_Knoll https://en.wikipedia.org/wiki/Erwin_Knoll

waterhouse 4y ago
Approximately the same phenomenon is also known as the Gell-Mann amnesia effect. It comes up on HN a lot, but for those who haven't encountered it before: > You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. > In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know. https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect
omnicognate 4y ago
It did the same thing for me on Gell-Mann amnesia, which is the same concept. Read carefully, it's a particularly good example of "wrong but highly convincing": > Gell-Mann amnesia is a term coined by science writer Michael Crichton to describe the phenomenon where people tend to forget information that contradicts their preconceptions or beliefs. The term is named after physicist Murray Gell-Mann, who won the 1969 Nobel Prize in Physics for his work on the theory of elementary particles. > According to Crichton, Gell-Mann amnesia occurs when people encounter information that contradicts their beliefs, but rather than updating their beliefs to reflect the new information, they simply forget the information and continue to hold onto their preconceptions. This type of cognitive bias can be a significant barrier to learning and can lead to flawed decision-making.
dinkumthinkum 4y ago
Exactly. We should be realistic about AI and ChatGPT and LLMs. You have great example of how “confidently” produced atrocious results even with the oft-cited ad hoc solution of “follow up” it gave even more misinformation and you, the user, only know this because you happened to know the answer. Oddly enough, if you google the types of things you are talking about, you will largely get the correct answer as the first result. There is something to be said about being naive about the potential of LLMs models but I am astounded how hyped and eager everyone is to hand over the future of all knowledge work to them without even clear evidence that that would make sense.

waterhouse 4y ago

Never tried it myself, but a question comes to mind: Can you ask it to state how confident it is in the truth of its answer? Is it well-calibrated in that sense? If so, perhaps that'd make it more useful.

RheingoldRiver 4y ago
A related thing I've done -- My mom's a college professor, and she's worried about her students using ChatGPT to cheat. So she wants to engineer questions in a way that cheating is impossible, perhaps by even incorporating ChatGPT into the curriculum in some ways. To explore possibilities, she's enlisting my help. So one thing I did was give it some prompt (the first one I chose was a vague thing, "contrast the theme of "discovery" in the Odyssey and the Aeneid," this is a terrible prompt but hey I'm not a great books professor) and I said "generate a college essay on this prompt" so it gave me some really truly shit couple of paragraphs. Then I said "that's not really college-level writing can you try again." And then I had it repeat this maybe 3 or 4 times. Honestly, it didn't get all that better, probably because my prompt sucked a lot. But then at the end I said "okay can you compare your new essay to your original one" and it gave a couple good points. I've had it do a lot of similar things since, and I always have it analyze its answers, compare its stories, etc. One HYSTERICAL answer that it gave was (and note that "The Great Migration" is a play that ChatGPT itself wrote itself in response to the prompt "can you write a play using the homeric tradition to talk about the great migration, as if you were august wilson"): > As a language model, I don't have personal preferences, it is not possible for me to say which play is better. Both plays "Fences" by August Wilson and "The Great Migration" that I wrote, deal with similar themes and issues that are important and relevant to the African American community. "Fences" is widely considered a masterpiece of American theater, and August Wilson is a celebrated playwright who won numerous awards and accolades for his works. "Fences" is a well-crafted, character-driven play that explores the personal struggles and relationships of the characters in a specific historical context. > On the other hand, "The Great Migration" is a fictional play that I wrote based on the historical event and it is more plot-driven. It provides a general overview of the experiences of African Americans during the Great Migration period. > Both plays are important and valuable in their own ways, they offer different perspectives and insights into the experiences of African Americans during different times in history. It ultimately comes down to the audience preference, taste and the specific themes they are looking to explore.

otabdeveloper4 4y ago

ChatGPT is automated copywriting, what did you expect?