11 ms·

Oceania Always at War with Eastasia: Generative AI and Knowledge Pollution

99 points by zdw 4y ago

deleted 4y ago
[deleted]
zitterbewegung 4y ago
When I was refining GPT-2 to make fake tweets of Donald Trump I encountered a similar problem about the DOI problem (and it would apply for anyone posting tweets with URLs). The twitter links would look legit because GPT-2 tried its best to create legitimate links. Here DOIs aren't being filtered by GPT3.5 so it does its best to generate a DOI that will look legitimate to a human to be correct but it will link to nothingness. So basically this is noise that GPT-3.5 needs to filter out.
aeternum 4y ago
Articles like this greatly underestimate the amount of misinformation out there throughout all of human history. Convincing people that make-believe is objective reality has been a cornerstone of human civilization. The pyramids would never be made without it, America would not have been founded: "We hold these truths to be self-evident". A superpower of humans is the ability to believe-in and spread these myths. Knowledge pollution existed much before AI. Did Indians actually scalp the Boston settlers? Nope, Benjamin Franklin made it up to strengthen anti-Indian sentiment.
- rootusrootus 4y ago
  Don't forget the largest of them all -- religion. Shared mythology seems to be hugely important to human society. If the mythology breaks down, it sure appears to take society right along with it. > America would not have been founded This is a great example, for sure. It's hard not to shed a little tear when you sit through the presentation they give on a US Capitol tour about the founding of the nation and the principles that govern it. Makes you want to stand right up and start reciting the pledge of allegiance, and wrap yourself in a glorious American flag. I'm only a little bit kidding. It really is a great story. I want it to be true, it makes me feel really good. It's a nice counterpoint to the Internet, which generally portrays America about as accurately but in the other direction. And I think that the breakdown of this shared mythology may be the most dangerous thing happening in America right now. Which isn't to say that the right-wing nationalist flag-waving rhetoric is the right answer, but we probably would be better off collectively if we all believed in the fairy tale of the founding.
  - pixl97 4y ago
    Society and culture are a shared mythology, without it you have a bunch of people doing their own thing but no real direction. Regions typically generated their own mythologies because of shared hardships. Religions typically co-oped those existing mythologies (though not always, genocide was used too) into their beliefs in order to more easily convert masses of fresh bodies. Successful religions typically have a few meta-memes built in. "Have kids or get punished". "Teach those kids your religion or get punished". "Tell other people to join your religion" "If those other people don't want to join your religion turn them into fertilizer". It's a very good way to spread a shared myth.
- pixl97 4y ago
  Eh, I think that's a slightly different problem though. First each of those problems was geographically limited and slow spreading. And yea, the king says dumb crap, but disagreeing with him will get your head chopped off. But at the same time people would do the things in the same way that ensured their continued survival in general. We are a long damned way from that world now. If you told me tomorrow that farmers decided to stop grown corn and decided to grow 'bio-bitcoin' this year I would not immediately discount that statement. Between consolidation in the industry and Americas short term obsession with profits over long term consequences, yea, it has some plausibility. This extreme consolidation in almost every market presents risk to societies at a global scale. The world back then was far more 'anti-fragile' than what exists today. And then you're not adding the constant firehose of bullshit and its effects. The bitrate of bullshit was far lower back then. Far back you didn't have books. To hear new and interesting bullshit you had to wait for travelers. Now there is a constant stream of bullshit from an unbound number of sources. It's not even like the older days where we stuck with a common bullshit narrative. We can each have our own custom made bullshit universe.
- VLM 4y ago
  "We never claimed the vaccine would prevent or slow the spread" We live in a civilization where there is propaganda, and only propaganda. Given that, ChatGPT will do just fine.
quantified 4y ago
> My wife’s career is in materials science, and she often writes technical papers related to her work. Recently she used ChatGPT to collect background information for a paper she was writing. Damning indictment of computer literacy, but a good example of the problem facing the majority. It's not that no one has been raising these issues, just read comments on HN posts. But no one in the hype-industrial complex has popularized them.
- xphilter 4y ago
  Agreed. Who the heck inserts a citation into a paper without actually reading the source material? Maybe ChatGPT is a great way to reveal the frauds.
  - elil17 4y ago
    Sounds like it got inserted into a draft. She did read the source material, just after that draft.
  - pietro72ohboy 4y ago
    Exactly. You're supposed to read and cite carefully, not just "citation dump" articles in favour of whatever argument you're trying to make.
    - deleted 4y ago
      [deleted]
    - burnished 4y ago
      My experience doing that is that this is not common practice - talking about following up on a reference to find it actually refutes a claim, or more commonly that a citation has little to do with what it is being cited for (more extrapolation than a completely inappropriate use). I agree with your should, but I do not think it is a realistic expectation right now.
      - falcor84 4y ago
aaroninsf 4y ago
Always remember when reasoning about, and making assertions about, what AI/ML will be capable of, and hence how it will impact existing conventions, institutions, disciplines, etc., to account for the fact that what is possible and commercial today, is characterized by non-linear patterns of change in any dimension you choose. More capable, cheaper, faster, everything. Consequently whenever you find yourself asserting what will be possible, or how we might be integrating, the systems of more than N months hence, consider that you may be not accounting for exponential and generational change, not to mention disruptive black swan events from external domains. TLDR modulo show-stopper from eg nuclear war, political dissolution, and climate crises, we are entering a moment of extraordinary disequilibrium, in which most of the world we inhabit will be upended and have to find new equilibrium.
- aaroninsf 4y ago
  In the case of this article, the application of my broader point would be, whatever we can say about the utility of LLM to generate intentional or unwitting disinformation, is already behind the curve. It's essentially a historical note. Example: Wikipedia has a position statement about LLM. It was done in a hurry by smart people. It's already woefully out of date. Things are moving too fast.
- jprete 4y ago
  I really don’t understand why people are so enthusiastic to cross the Rubicon without even looking.
masukomi 4y ago
I've said it before, but it needs repeating. chatGPT is a very confident FICTION generator. Any facts it produces are purely coincidental. Please stop assuming anything it says it true. It was never designed to do that and it provably doesn't do that.
- KineticLensman 4y ago
  A description I saw here a couple of days ago was ‘blarney generator’. I think that captures it perfectly
- haolez 4y ago
  I'm using it for summarization and classification of big texts and it's showing a hit-ratio bigger than human analysts. I'm satisfied with it as is :)
  - vidarh 4y ago
    I'm ambivalent. It summarises text on papers that are simple enough, but interestingly it fails in ways that suggests it sometimes fails because it isn't "smart enough." E.g. I asked it about "Code Generation on the Fly: A Key to Portable Software", the doctoral dissertation of Professor Michel Franz, and a paper I know very well, and it spit back a description of the process that was perhaps superficially correct but the kind of correct you'd expect of a lazy student who'd not bothered to actually understand the paper. The summary was largely meaningless twaddle that didn't actually tell you anything useful. I then started probing it for more detail, and pointing out mistaken assumptions, and it got better. But it was first when I pointed out that the technique for encoding and decoding has a resemblance to Lempel-Ziv-Welch compression that it succeeded in identifying the key part of the paper and give a description that while still not quite accurate at least captured the essence of the method. It felt like I was giving an exam on the topic rather than "getting help" summarising it, and if I didn't know the paper, it would've been hard to tell if I was getting close without reading it. For conceptually simpler things, though, I've gotten great results, and it seems to work great for things where I know what a good result looks like ahead of time, and "just" want to save looking up details where I'll recognise the right thing when I see it but don't remember it by heart. That said, to be able to point out the resemblance to LZW and have it connect the dots is at the same time fairly impressive - I've had to point that out to experienced developers (the paper mentions Welch, "This method bears some resemblance to commonly used data compression schemes [Wel84]", with Wel84 reading: "T. A. Welch; A Technique for High_Performance Data Compression; IEEE Computer, 17:6, 8-19; 1984. {2}" but it does not go into any further detail of how they relate). I think, overall, that this is an indication that it will get better at this as the model size increases further. It was able to at least recognise the semantic similarity or identify that this similiarity has been described elsewhere and use it to guide it's response in a way that strikes me as far from trivial. [It did go on to hallucinate several relevant papers on the subject; interestingly it confirmed it had made them up when I asked if it had]
yrdmb 4y ago
> There’s potential danger when pranksters alter Wikipedia entries, or fraudsters publish scientific papers with bogus data, but the truth eventually comes out. Pranksters? No, the danger is the people with power meddling with information. They start wars, not pranksters. But there's nothing that can be done about that so it's pointless talking about it. > The scientific-historical record of accumulating human knowledge has grown steadily for ten thousand years. Yes some information gets lost, some gets proven wrong, some is disputed, and some gets hidden when winners spin the facts to flatter themselves. "Scientific-historical"? Almost all historical information ( written record ) is lost. Less than 1% of all written text from ancient greece and rome survived. Also it's not a matter of the winners hiding facts that's the issue, it's the winners pushing particular "facts". Take the civil war. It was a war where one side had slave states and the other side was the confederacy. Yet we are told that it was a war to end slavery. Have fun squaring that circle. > But despite the system’s flaws, until now it’s worked fairly well to maintain our shared understanding about what’s real and what’s true. Our understanding of what's real and true is solely a function of who is in power. Same as it always has been and always will be. It doesn't matter whether the tool is spoken language, written language, printing press, radio, tv, social media or AI. Knowledge "pollution" is what societies and nations breath to survive. The only struggle is the fight over who gets to pollute.
- crimjow 4y ago
  [flagged]
  - yrdmb 4y ago
    [flagged]
    - mrtranscendence 4y ago
      Are you denying that slavery was a factor that led to the civil war? Because that's incontrovertible. At the very least I can't see how you can credibly disagree that the issue of the spread of slavery was one driving factor. Pointing to Kentucky and Delaware misses the point. It's not like the thirteenth amendment -- ratified in the same year the war ended, which is not coincidental -- didn't apply to them.
EGreg 4y ago
You know, I used to think George Orwell's 1984 was fanciful, but then I saw our governments and media do it all the time, including in current and past conflicts: Ukraine: https://www.youtube.com/watch?v=fvsnYJPqLSI https://www.youtube.com/watch?v=fvsnYJPqLSI Afghanistan: https://www.outlookindia.com/website/story/how-jimmy-carter-and-i-started-the-mujahideen/213722 https://www.outlookindia.com/website/story/how-jimmy-carter-... Q: When the Soviets justified their intervention by asserting that they intended to fight against a secret involvement of the United States in Afghanistan, people didn't believe them. However, there was a basis of truth. You don't regret anything today? Brzezinski: Regret what? That secret operation was an excellent idea. It had the effect of drawing the Russians into the Afghan trap and you want me to regret it? The day that the Soviets officially crossed the border, I wrote to President Carter: We now have the opportunity of giving to the USSR its Vietnam war. Indeed, for almost 10 years, Moscow had to carry on a war unsupportable by the government, a conflict that brought about the demoralization and finally the breakup of the Soviet empire.
- rootusrootus 4y ago
  I can't watch videos from where I'm at, so I'm curious about the Ukraine claim. Am I supposed to believe that Russia did not in fact invade Ukraine? It's all just a bit of propaganda? Or are you claiming that we somehow convinced them it was a great idea and thus it is somehow our fault they decided to roll those tanks?
  - EGreg 4y ago
    It’s basically a video where the BBC claims the exact opposite of what they were reporting before, and denies the very things they repeatedly said before, until the UK government got actively involved and influenced them to change their narrative: 2014 BBC for example: https://m.youtube.com/watch?v=5SBo0akeDMY https://m.youtube.com/watch?v=5SBo0akeDMY During 2014 Maidan revolution: “There are far right neo nazis in Ukraine, they are the most organized and active part of the revolution, here is a whole documentary about that where we go interview them about their goals.” 2017, 2018, 2019: “Paramilitary groups like Azov and Dnipro etc. contain many far-right nationalist fighters, about 20% admit to holding neo nazi beliefs”: https://www.haaretz.com/israel-news/2018-07-09/ty-article/rights-groups-demand-israel-stop-arming-neo-nazis-in-the-ukraine/0000017f-e080-d7b2-a77f-e3870e1c0000 https://www.haaretz.com/israel-news/2018-07-09/ty-article/ri... And then 2022: “Azov has hardly any far right, they are in fact just a part of the official armed forces of Ukraine, and it is the Russians committing all the atrocities in battle”: https://www.bbc.com/news/av/world-europe-61205685 https://www.bbc.com/news/av/world-europe-61205685 I am sure there is truth on all sides but certainly a wide gulf between what they reported then and now. Here in USA, Trump supporters could also believe new Trump and even though he contradicted old Trump totally. Trump on video side by side: https://youtu.be/zitZGDJcHAU https://youtu.be/zitZGDJcHAU People just get fervor and overlook that stuff
CmdrLoskene 4y ago
Can the proliferation of LLMs in the hands of bad actors precipitate a society-wide epistemic crisis? Survey says "maybe".
- rootusrootus 4y ago
  I expect a non-trivial number of people will tell you that we've been in a full blown epistemic crisis for some years now, and we have yet to find out just how far the rabbit hole will go.
  - williamcotton 4y ago
    The epistemic crisis mirrors the proliferation of electronic media. It really ramped up with the common adoption of the smartphone and social media.
williamcotton 4y ago
Information != Knowledge We can have information pollution but knowledge pollution is incoherent.
- deleted 4y ago
  [deleted]
cjbgkagh 4y ago
Just today I read a Quora bot answer that was confidently wrong, I could tell as I already knew the correct answer. The bot answer also contradicted the unanimous human answers that predated it, so they couldn’t even be bothered do a basic sanity check before posting the bot answer.
undersuit 4y ago
>They’re now in the process of changing the journals’ editorial controls to address this risk, but it’s unclear how far down the rabbit hole one must go checking citations’ citations ad infinitum. This reminds me of a project my coworker did for the university cafeteria. They wanted a system so that their website could list all the ingredients of foods that were going to be served. My coworker was stuck using XSLT to make the page and he decided to use 6-layer nested if instead of learning how to use recursion in XSLT. We had to rewrite the system from scratch and were able to get rid of this limitation on the database side at which we discovered that some recipes had 7 and 8 deep graphs and only looking 6 deep was hiding known allergens. It's good they're considering how extensively you should verify the info. So all the cited papers in the submitted article are relevant and authentic... are all the cited papers in the cited papers in the submitted article relevant and authentic? Big fun for the reviewers.
phailhaus 4y ago
Companies are lying about what these LLMs are capable of, full stop. That is the problem. And the reason for that is because they can't justify their costs if all they've created is a really good chat bot and writing assistant. The responsible thing to do would be to have these LLMs refuse to respond if asked informational questions, but that would kill interest. So they won't. They're going to keep trying to shove it into applications that could potentially generate recurring revenue, and it's going to keep failing. I suspect they will go the way of voice assistants.
- falcor84 4y ago
  Why should we hold AIs to a higher standard than humans. The internet (and the world at large) is full of people confidently spewing misinformation - should we go ahead and censor all of them?
  - phailhaus 4y ago
    Huh? What I'm telling you is that this is not "AI", it is a LANGUAGE MODEL. It models language. People literally think that these chatbots have access to some sort of "knowledge base" or are capable of "looking up information", but they can't. It's not a matter of "misinformation", these companies are trying to trick you into thinking they have something that they do not. And the idea that we shouldn't hold our tools to a higher standard is a joke. It's like if you sold me a screwdriver as a hammer and then went "wow, nobody's perfect, ok so we should just BAN all humans that are bad at hammering?!"
    - falcor84 4y ago
      >...this is not "AI", it is a LANGUAGE MODEL. It sounds like we have a semantic disagreement here. As I understand it, a language model is a mathematical construct, while a particular implementation of a language model as a software system is an AI. At the least it's as much an AI as a Chess Engine, right? > It's like if you sold me a screwdriver as a hammer and then went "wow, nobody's perfect, ok so we should just BAN all humans that are bad at hammering?!" That doesn't sound quite analogous to me. My scenario would be that for a particular hammering job, I can hire either a human or a hammering AI. If the hammering AI is imperfect, but still better and/or cheaper than the typical human worker, (at least as a cold-hearted business manager) I'd prefer the AI.
MWil 4y ago
"What’s fascinating to me is that as recently as a year ago, virtually nobody was talking about this risk. It wasn’t on anybody’s radar." Like-minded people like Carl Malamud have worried about this in the context of legal knowledge for over a decade. When there is a monopoly (duopoly of West/Lexis(Thompson Reuters) on the "official" publication of statutes, regulations, codes, and other aspects of government, what protections are there that they won't introduce inaccuracies of their own? What was really scary was that in order to contract for a new copy of say, a certain state's set of statutes, you used to (maybe you still do, I don't know) have to agree to either send back to the company or destroy your old copies. You couldn't be allowed to compare the version from the past if you thought "well this is weird, I always thought murder was illegal but this version says murder is legal." You don't know if that's a typo in this one version, a cosmic ray bitflip, or something more nefarious (and murder would probably be the least conspicious law to get that accomplished). What if something that used to be punishable by a $10,000 or prison has a new version that suddenly states $100,000 or prison. We have to have a way that preserves all old legal knowledge and all new legal knowledge - it will never be safe to just go all on digital infrastructure in this regard. There are so many times as a lawyer I want to see the oldest existing paper copy. And now I have to worry about AI hallucinating an authentic-looking paper copy...

99 points by zdw 4y ago

deleted 4y ago

[deleted]

When I was refining GPT-2 to make fake tweets of Donald Trump I encountered a similar problem about the DOI problem (and it would apply for anyone posting tweets with URLs). The twitter links would look legit because GPT-2 tried its best to create legitimate links. Here DOIs aren't being filtered by GPT3.5 so it does its best to generate a DOI that will look legitimate to a human to be correct but it will link to nothingness. So basically this is noise that GPT-3.5 needs to filter out.

aeternum 4y ago

Articles like this greatly underestimate the amount of misinformation out there throughout all of human history. Convincing people that make-believe is objective reality has been a cornerstone of human civilization. The pyramids would never be made without it, America would not have been founded: "We hold these truths to be self-evident". A superpower of humans is the ability to believe-in and spread these myths. Knowledge pollution existed much before AI. Did Indians actually scalp the Boston settlers? Nope, Benjamin Franklin made it up to strengthen anti-Indian sentiment.

rootusrootus 4y ago
Don't forget the largest of them all -- religion. Shared mythology seems to be hugely important to human society. If the mythology breaks down, it sure appears to take society right along with it. > America would not have been founded This is a great example, for sure. It's hard not to shed a little tear when you sit through the presentation they give on a US Capitol tour about the founding of the nation and the principles that govern it. Makes you want to stand right up and start reciting the pledge of allegiance, and wrap yourself in a glorious American flag. I'm only a little bit kidding. It really is a great story. I want it to be true, it makes me feel really good. It's a nice counterpoint to the Internet, which generally portrays America about as accurately but in the other direction. And I think that the breakdown of this shared mythology may be the most dangerous thing happening in America right now. Which isn't to say that the right-wing nationalist flag-waving rhetoric is the right answer, but we probably would be better off collectively if we all believed in the fairy tale of the founding.
- pixl97 4y ago
  Society and culture are a shared mythology, without it you have a bunch of people doing their own thing but no real direction. Regions typically generated their own mythologies because of shared hardships. Religions typically co-oped those existing mythologies (though not always, genocide was used too) into their beliefs in order to more easily convert masses of fresh bodies. Successful religions typically have a few meta-memes built in. "Have kids or get punished". "Teach those kids your religion or get punished". "Tell other people to join your religion" "If those other people don't want to join your religion turn them into fertilizer". It's a very good way to spread a shared myth.
pixl97 4y ago
Eh, I think that's a slightly different problem though. First each of those problems was geographically limited and slow spreading. And yea, the king says dumb crap, but disagreeing with him will get your head chopped off. But at the same time people would do the things in the same way that ensured their continued survival in general. We are a long damned way from that world now. If you told me tomorrow that farmers decided to stop grown corn and decided to grow 'bio-bitcoin' this year I would not immediately discount that statement. Between consolidation in the industry and Americas short term obsession with profits over long term consequences, yea, it has some plausibility. This extreme consolidation in almost every market presents risk to societies at a global scale. The world back then was far more 'anti-fragile' than what exists today. And then you're not adding the constant firehose of bullshit and its effects. The bitrate of bullshit was far lower back then. Far back you didn't have books. To hear new and interesting bullshit you had to wait for travelers. Now there is a constant stream of bullshit from an unbound number of sources. It's not even like the older days where we stuck with a common bullshit narrative. We can each have our own custom made bullshit universe.
VLM 4y ago
"We never claimed the vaccine would prevent or slow the spread" We live in a civilization where there is propaganda, and only propaganda. Given that, ChatGPT will do just fine.

quantified 4y ago

> My wife’s career is in materials science, and she often writes technical papers related to her work. Recently she used ChatGPT to collect background information for a paper she was writing. Damning indictment of computer literacy, but a good example of the problem facing the majority. It's not that no one has been raising these issues, just read comments on HN posts. But no one in the hype-industrial complex has popularized them.

xphilter 4y ago
Agreed. Who the heck inserts a citation into a paper without actually reading the source material? Maybe ChatGPT is a great way to reveal the frauds.
- elil17 4y ago
  Sounds like it got inserted into a draft. She did read the source material, just after that draft.
- pietro72ohboy 4y ago
  Exactly. You're supposed to read and cite carefully, not just "citation dump" articles in favour of whatever argument you're trying to make.
  - deleted 4y ago
    [deleted]
  - burnished 4y ago
    My experience doing that is that this is not common practice - talking about following up on a reference to find it actually refutes a claim, or more commonly that a citation has little to do with what it is being cited for (more extrapolation than a completely inappropriate use). I agree with your should, but I do not think it is a realistic expectation right now.
    - falcor84 4y ago

aaroninsf 4y ago

Always remember when reasoning about, and making assertions about, what AI/ML will be capable of, and hence how it will impact existing conventions, institutions, disciplines, etc., to account for the fact that what is possible and commercial today, is characterized by non-linear patterns of change in any dimension you choose. More capable, cheaper, faster, everything. Consequently whenever you find yourself asserting what will be possible, or how we might be integrating, the systems of more than N months hence, consider that you may be not accounting for exponential and generational change, not to mention disruptive black swan events from external domains. TLDR modulo show-stopper from eg nuclear war, political dissolution, and climate crises, we are entering a moment of extraordinary disequilibrium, in which most of the world we inhabit will be upended and have to find new equilibrium.

aaroninsf 4y ago
In the case of this article, the application of my broader point would be, whatever we can say about the utility of LLM to generate intentional or unwitting disinformation, is already behind the curve. It's essentially a historical note. Example: Wikipedia has a position statement about LLM. It was done in a hurry by smart people. It's already woefully out of date. Things are moving too fast.
jprete 4y ago
I really don’t understand why people are so enthusiastic to cross the Rubicon without even looking.

masukomi 4y ago

I've said it before, but it needs repeating. chatGPT is a very confident FICTION generator. Any facts it produces are purely coincidental. Please stop assuming anything it says it true. It was never designed to do that and it provably doesn't do that.

KineticLensman 4y ago
A description I saw here a couple of days ago was ‘blarney generator’. I think that captures it perfectly
haolez 4y ago
I'm using it for summarization and classification of big texts and it's showing a hit-ratio bigger than human analysts. I'm satisfied with it as is :)
- vidarh 4y ago
  I'm ambivalent. It summarises text on papers that are simple enough, but interestingly it fails in ways that suggests it sometimes fails because it isn't "smart enough." E.g. I asked it about "Code Generation on the Fly: A Key to Portable Software", the doctoral dissertation of Professor Michel Franz, and a paper I know very well, and it spit back a description of the process that was perhaps superficially correct but the kind of correct you'd expect of a lazy student who'd not bothered to actually understand the paper. The summary was largely meaningless twaddle that didn't actually tell you anything useful. I then started probing it for more detail, and pointing out mistaken assumptions, and it got better. But it was first when I pointed out that the technique for encoding and decoding has a resemblance to Lempel-Ziv-Welch compression that it succeeded in identifying the key part of the paper and give a description that while still not quite accurate at least captured the essence of the method. It felt like I was giving an exam on the topic rather than "getting help" summarising it, and if I didn't know the paper, it would've been hard to tell if I was getting close without reading it. For conceptually simpler things, though, I've gotten great results, and it seems to work great for things where I know what a good result looks like ahead of time, and "just" want to save looking up details where I'll recognise the right thing when I see it but don't remember it by heart. That said, to be able to point out the resemblance to LZW and have it connect the dots is at the same time fairly impressive - I've had to point that out to experienced developers (the paper mentions Welch, "This method bears some resemblance to commonly used data compression schemes [Wel84]", with Wel84 reading: "T. A. Welch; A Technique for High_Performance Data Compression; IEEE Computer, 17:6, 8-19; 1984. {2}" but it does not go into any further detail of how they relate). I think, overall, that this is an indication that it will get better at this as the model size increases further. It was able to at least recognise the semantic similarity or identify that this similiarity has been described elsewhere and use it to guide it's response in a way that strikes me as far from trivial. [It did go on to hallucinate several relevant papers on the subject; interestingly it confirmed it had made them up when I asked if it had]

yrdmb 4y ago

> There’s potential danger when pranksters alter Wikipedia entries, or fraudsters publish scientific papers with bogus data, but the truth eventually comes out. Pranksters? No, the danger is the people with power meddling with information. They start wars, not pranksters. But there's nothing that can be done about that so it's pointless talking about it. > The scientific-historical record of accumulating human knowledge has grown steadily for ten thousand years. Yes some information gets lost, some gets proven wrong, some is disputed, and some gets hidden when winners spin the facts to flatter themselves. "Scientific-historical"? Almost all historical information ( written record ) is lost. Less than 1% of all written text from ancient greece and rome survived. Also it's not a matter of the winners hiding facts that's the issue, it's the winners pushing particular "facts". Take the civil war. It was a war where one side had slave states and the other side was the confederacy. Yet we are told that it was a war to end slavery. Have fun squaring that circle. > But despite the system’s flaws, until now it’s worked fairly well to maintain our shared understanding about what’s real and what’s true. Our understanding of what's real and true is solely a function of who is in power. Same as it always has been and always will be. It doesn't matter whether the tool is spoken language, written language, printing press, radio, tv, social media or AI. Knowledge "pollution" is what societies and nations breath to survive. The only struggle is the fight over who gets to pollute.

crimjow 4y ago
[flagged]
- yrdmb 4y ago
  [flagged]
  - mrtranscendence 4y ago
    Are you denying that slavery was a factor that led to the civil war? Because that's incontrovertible. At the very least I can't see how you can credibly disagree that the issue of the spread of slavery was one driving factor. Pointing to Kentucky and Delaware misses the point. It's not like the thirteenth amendment -- ratified in the same year the war ended, which is not coincidental -- didn't apply to them.

EGreg 4y ago

You know, I used to think George Orwell's 1984 was fanciful, but then I saw our governments and media do it all the time, including in current and past conflicts: Ukraine: https://www.youtube.com/watch?v=fvsnYJPqLSI https://www.youtube.com/watch?v=fvsnYJPqLSI Afghanistan: https://www.outlookindia.com/website/story/how-jimmy-carter-and-i-started-the-mujahideen/213722 https://www.outlookindia.com/website/story/how-jimmy-carter-... Q: When the Soviets justified their intervention by asserting that they intended to fight against a secret involvement of the United States in Afghanistan, people didn't believe them. However, there was a basis of truth. You don't regret anything today? Brzezinski: Regret what? That secret operation was an excellent idea. It had the effect of drawing the Russians into the Afghan trap and you want me to regret it? The day that the Soviets officially crossed the border, I wrote to President Carter: We now have the opportunity of giving to the USSR its Vietnam war. Indeed, for almost 10 years, Moscow had to carry on a war unsupportable by the government, a conflict that brought about the demoralization and finally the breakup of the Soviet empire.

rootusrootus 4y ago
I can't watch videos from where I'm at, so I'm curious about the Ukraine claim. Am I supposed to believe that Russia did not in fact invade Ukraine? It's all just a bit of propaganda? Or are you claiming that we somehow convinced them it was a great idea and thus it is somehow our fault they decided to roll those tanks?
- EGreg 4y ago
  It’s basically a video where the BBC claims the exact opposite of what they were reporting before, and denies the very things they repeatedly said before, until the UK government got actively involved and influenced them to change their narrative: 2014 BBC for example: https://m.youtube.com/watch?v=5SBo0akeDMY https://m.youtube.com/watch?v=5SBo0akeDMY During 2014 Maidan revolution: “There are far right neo nazis in Ukraine, they are the most organized and active part of the revolution, here is a whole documentary about that where we go interview them about their goals.” 2017, 2018, 2019: “Paramilitary groups like Azov and Dnipro etc. contain many far-right nationalist fighters, about 20% admit to holding neo nazi beliefs”: https://www.haaretz.com/israel-news/2018-07-09/ty-article/rights-groups-demand-israel-stop-arming-neo-nazis-in-the-ukraine/0000017f-e080-d7b2-a77f-e3870e1c0000 https://www.haaretz.com/israel-news/2018-07-09/ty-article/ri... And then 2022: “Azov has hardly any far right, they are in fact just a part of the official armed forces of Ukraine, and it is the Russians committing all the atrocities in battle”: https://www.bbc.com/news/av/world-europe-61205685 https://www.bbc.com/news/av/world-europe-61205685 I am sure there is truth on all sides but certainly a wide gulf between what they reported then and now. Here in USA, Trump supporters could also believe new Trump and even though he contradicted old Trump totally. Trump on video side by side: https://youtu.be/zitZGDJcHAU https://youtu.be/zitZGDJcHAU People just get fervor and overlook that stuff

CmdrLoskene 4y ago

Can the proliferation of LLMs in the hands of bad actors precipitate a society-wide epistemic crisis? Survey says "maybe".

rootusrootus 4y ago
I expect a non-trivial number of people will tell you that we've been in a full blown epistemic crisis for some years now, and we have yet to find out just how far the rabbit hole will go.
- williamcotton 4y ago
  The epistemic crisis mirrors the proliferation of electronic media. It really ramped up with the common adoption of the smartphone and social media.

williamcotton 4y ago

Information != Knowledge We can have information pollution but knowledge pollution is incoherent.

deleted 4y ago
[deleted]

cjbgkagh 4y ago

Just today I read a Quora bot answer that was confidently wrong, I could tell as I already knew the correct answer. The bot answer also contradicted the unanimous human answers that predated it, so they couldn’t even be bothered do a basic sanity check before posting the bot answer.

undersuit 4y ago

>They’re now in the process of changing the journals’ editorial controls to address this risk, but it’s unclear how far down the rabbit hole one must go checking citations’ citations ad infinitum. This reminds me of a project my coworker did for the university cafeteria. They wanted a system so that their website could list all the ingredients of foods that were going to be served. My coworker was stuck using XSLT to make the page and he decided to use 6-layer nested if instead of learning how to use recursion in XSLT. We had to rewrite the system from scratch and were able to get rid of this limitation on the database side at which we discovered that some recipes had 7 and 8 deep graphs and only looking 6 deep was hiding known allergens. It's good they're considering how extensively you should verify the info. So all the cited papers in the submitted article are relevant and authentic... are all the cited papers in the cited papers in the submitted article relevant and authentic? Big fun for the reviewers.

phailhaus 4y ago

Companies are lying about what these LLMs are capable of, full stop. That is the problem. And the reason for that is because they can't justify their costs if all they've created is a really good chat bot and writing assistant. The responsible thing to do would be to have these LLMs refuse to respond if asked informational questions, but that would kill interest. So they won't. They're going to keep trying to shove it into applications that could potentially generate recurring revenue, and it's going to keep failing. I suspect they will go the way of voice assistants.

falcor84 4y ago
Why should we hold AIs to a higher standard than humans. The internet (and the world at large) is full of people confidently spewing misinformation - should we go ahead and censor all of them?
- phailhaus 4y ago
  Huh? What I'm telling you is that this is not "AI", it is a LANGUAGE MODEL. It models language. People literally think that these chatbots have access to some sort of "knowledge base" or are capable of "looking up information", but they can't. It's not a matter of "misinformation", these companies are trying to trick you into thinking they have something that they do not. And the idea that we shouldn't hold our tools to a higher standard is a joke. It's like if you sold me a screwdriver as a hammer and then went "wow, nobody's perfect, ok so we should just BAN all humans that are bad at hammering?!"
  - falcor84 4y ago
    >...this is not "AI", it is a LANGUAGE MODEL. It sounds like we have a semantic disagreement here. As I understand it, a language model is a mathematical construct, while a particular implementation of a language model as a software system is an AI. At the least it's as much an AI as a Chess Engine, right? > It's like if you sold me a screwdriver as a hammer and then went "wow, nobody's perfect, ok so we should just BAN all humans that are bad at hammering?!" That doesn't sound quite analogous to me. My scenario would be that for a particular hammering job, I can hire either a human or a hammering AI. If the hammering AI is imperfect, but still better and/or cheaper than the typical human worker, (at least as a cold-hearted business manager) I'd prefer the AI.

MWil 4y ago

"What’s fascinating to me is that as recently as a year ago, virtually nobody was talking about this risk. It wasn’t on anybody’s radar." Like-minded people like Carl Malamud have worried about this in the context of legal knowledge for over a decade. When there is a monopoly (duopoly of West/Lexis(Thompson Reuters) on the "official" publication of statutes, regulations, codes, and other aspects of government, what protections are there that they won't introduce inaccuracies of their own? What was really scary was that in order to contract for a new copy of say, a certain state's set of statutes, you used to (maybe you still do, I don't know) have to agree to either send back to the company or destroy your old copies. You couldn't be allowed to compare the version from the past if you thought "well this is weird, I always thought murder was illegal but this version says murder is legal." You don't know if that's a typo in this one version, a cosmic ray bitflip, or something more nefarious (and murder would probably be the least conspicious law to get that accomplished). What if something that used to be punishable by a $10,000 or prison has a new version that suddenly states $100,000 or prison. We have to have a way that preserves all old legal knowledge and all new legal knowledge - it will never be safe to just go all on digital infrastructure in this regard. There are so many times as a lawyer I want to see the oldest existing paper copy. And now I have to worry about AI hallucinating an authentic-looking paper copy...