12 ms·

AI and the Problem of Knowledge Collapse

189 points by kmdupree 3y ago

resolutebat 3y ago
So by this definition, do we already have "knowledge collapse" by Wikipedia? Because if you search for a random concept, that's usually the first hit, and it's also what countless other sources draw on.
- rwbt 3y ago
  Yes, we do kind of.
  - hprotagonist 3y ago
    and i distinctly remember this critique being made at wikipedia's advent, as well. and it is not without justification! https://undark.org/2021/08/12/wikipedia-has-a-language-problem-heres-how-to-fix-it/ https://undark.org/2021/08/12/wikipedia-has-a-language-probl...
- nickpsecurity 3y ago
  The same warning was given for Google. Except those people added that it would reduce problem solving ability, too. People would get used to whatever simple, instant content rose to the top. They'd gradually lose some or all of their ability to figure out the same things on their own. One submission here was a tech guy at a school saying that was already happening where he worked.
  - aoanla 3y ago
    I mean, it does - people search stuff all the time now, rather than thinking about it.
    - 082349872349872 3y ago
      IIRC, that was Socrates' complaint to Phaedrus about writing: that reading (because it was "high tech" at the time?) led only to an illusion of understanding. Elsewhere Phaedrus echoes with a very modern complaint (even though search engines wouldn't arrive for another 2'300 years): They would say in reply that he is a madman or a pedant who fancies that he is a physician because he has read something in a book, or has stumbled on a prescription or two, although he has no real understanding of the art of medicine. https://www.gutenberg.org/files/1636/1636-h/1636-h.htm https://www.gutenberg.org/files/1636/1636-h/1636-h.htm
      - jprete 3y ago
        Socrates wasn't wrong. Reading a lot gives you a partial understanding but it isn't complete without experiencing the thing for yourself. Arguably the Internet is the natural home of authoritatively stated but uninformed opinions - the exact result of reading a lot about a subject without having any experience of it.
        contravariant 3y ago
        Ironically we can only know he was right by making the exact mistake he was warning us against.
      - voidhorse 3y ago
        I think the Phaedrus is all about the importance of practice. I've been reading a lot of math books lately, but I don't actually grok anything well until I sit down and try to reason through the material myself, write my own little proofs, try to deconstruct what's being said actively with pen and paper. Similarly, I understand a work of literature far more deeply if I take active notes, and/or write a small essay about my interpretation. I become a better writer by reading good writers and emulating them in my own writing practice. Writing was a threat to poets when the goal was still to recite a compelling live performance, which, to do this well, would require memorization and practice—today, still, we ask that actors do not have paper scripts in front of them when performing in a film. This is kind of the threat that tools like LLM's pose. Their power to generate decent results means that far more people will eschew practice for "good enough" LLM produced results. Creation will become even more transactional, and (many) people will quickly fail to "see the point" in practicing until we have a culture that's degraded even further than it already has today.
- AlienRobot 3y ago
  Personally, I think the problem is that people abuse Google for things it's really not designed to do, and they don't even realize that. Google is great at finding official webpages by their exact title. If you type the title of a news headline from the 90s, Google will give you the link to it. I think that is amazing. Basically anything that has a canonical URL, Google is good at finding. But when you search for "how to do X" for example, there will be several results that are perfectly valid and they will still need to be ranked in a list. Because it's not a "list" of results, it's ranking of relevancy. So to avoid showing spam, Google will push to the top websites it finds trustworthy. And now every top result comes from the same website. If you need an explanation for the Xz incident, for example, there is no canonical URL for it. There will be several news websites, youtube channels, etc. that have talked about it, competing to be the top result. Google still has to rank them even though the algorithm can't tell fact apart from parody, so no matter what Google does, Google will be the one judging which content most people will read when they want to know about a certain topic. To borrow my fellow robot's words, people are finding knowledge through an algorithmically curated aperture: Google's SERP. If they're evil, they have the power to control everyone on Earth. If they're good, they must be going insane with what to do with their users' crippling dependency on them as a source of truth.
  - 082349872349872 3y ago
    I find Google isn't so good anymore for finding things by title; rather than being a search engine they are slowly becoming more like a politician answering reporters' questions, in that instead of returning results based on the terms I asked for, they insist on returning results for the terms they believe I should have asked for.
    - AlienRobot 3y ago
      Can you give me examples? Personally, I don't think I've ever had a problem finding things by their name or title on Google. If I type python, I get python's website, if I type hacker news, I get this website. I can type the name of an old program and get their webpage on sourceforge for example. I've also tested that it works with finding the official websites of celebrities, e.g. if you type eminem, you get his official website. It does seem to have biases toward recency and product pages, but finding things by their title isn't a thing Google is bad at.
- the_af 3y ago
  Probably Wikipedia is an early precursor of this problem, yes. It may not be entirely new, but AI might accelerate it to really harmful speeds. Just pondering this, of course.
JieJie 3y ago
The discussion section is quite illuminating. "While much recent attention has been on the problem of LLMs misleadingly presenting fiction as fact (hallucination), this may be less of an issue than the problem of representativeness across a distribution of possible responses. Hallucination of verifiable, concrete facts is often easy to correct for. Yet many real world questions do not have well-defined, verifiably true and false answers. If a user asks, for example, “What causes inflation?” and a LLM answers “monetary policy”, the problem isn’t one of hallucination, but of the failure to reflect the full-distribution of possible answers to the question, or at least provide an overview of the main schools of economic thought."
- ben_w 3y ago
  First thought: Oh no, they want LLMs to be even more vocal about nuance Second thought: People aren't going to read nuance Third thought: They should Fourth thought: Have you met people? They'll get angry with you for even suggesting it
- voidhorse 3y ago
  Right. It all hooks up to a naive, popular philosophy of knowledge that I think has been burgeoning since the birth of search engines: people today tend to think that everything is decomposable into atomic facts and that all questions have concrete, singular, atomic answers. Obviously, this could not be further from the truth, but I think ever since the advent of things like "answer cards" in search, people have been led into this technological bias trap of thinking that whatever answer the engine spits out must be the canonical answer and forget that we still have a broad range of questions and research areas we don't have anything close to definitive answers on. The problem is only exacerbated when people make this mistake in domains that will never have singular answers like politics and ethics.
knowsuchagency 3y ago
I feel like this has always been the case. The entire information economy is based on a few key publishers and figures. You see it in news, academia, social media -- there's orthodoxy everywhere. Not sure how AI is any different.
- iraqmtpizza 3y ago
  In the 1990s people read their town's newspaper. Now people in Arizona read the Daily Mail
  - 48864w6ui 3y ago
    In the 1990s people who wanted to advertise in that town had to do so in local media. Now they can ad tech and the Daily Mail will arrange for it to be served.
  - simonw 3y ago
    I worked for a local newspaper in Kansas around 2003/2004 and one thing I found surprising was that journalists there were frequently on the hook for writing up national stories - things that would come in off the wire services and then be re-written for the local audience.
    - iraqmtpizza 3y ago
      Did they write it like What in tarnation is going on in the Big Apple?
- the_af 3y ago
  It's probably different in that the few key publishers reached few people, but now AI is "sold" as for everyone and it's not unreasonable to expect it to become consumerized into a magic box of information, news and problem solving in the near future. It may not be a fundamentally new process, but what if it becomes so ubiquitous and fast that it becomes really harmful? And would there be a point of no return for society if this happened?
- hackerlight 3y ago
karaterobot 3y ago
> Informally, we define knowledge collapse as the progressive narrowing over time (or over technological representations) of the set of information available to humans, along with a concomitant narrowing in the perceived availability and utility of different sets of information. > The main focus of the model is whether individuals decide to invest in innovation or learning ... in the ‘traditional’ way, through a possibly cheaper AI-enabled process, or not at all. The idea is to capture, for example, the difference between someone who does extensive research in an archive rather than just relying on readily-available materials, or someone who takes the time to read a full book rather than reading a two-paragraph LLM-generated summary. > Under these conditions, excessive reliance on AI-generated content over time leads to a curtailing of the eccentric and rare viewpoints that maintain a comprehensive vision of the world. My intuition is that AI will just accelerate the trends that the internet brought on, which is that eccentric viewpoints are actually pretty common, even ones based on research and in fact. The internet people mostly use has become relatively generic, consumed through a pretty narrow, curated aperture (social media). This feels analogous to getting it through AI, as described in the article. Yet, people are still learning about eccentric, marginal stuff all the time, especially compared to, say, 50 years ago. Assuming the AI's responses aren't artificially limited, people who are interested enough to look will still get to learn about topics in the long tail of the distribution, even in a world of ubiquitous AI. And they'll be able to dive as deeply into them as they do today. I'm not really worried about that. If anything, the knowledge collapse will be at the center. Basic liberal education topics are what will go away. Or rather, they will be offloaded to AI. In the same way that people say they don't need to learn arithmetic because they have a calculator, my guess is people will be more likely to decide not to worry about what previous generations considered core knowledge: history, geography, the canon, and so on. "I don't have to know it, I can look it up". That'll all go away even faster than it's going now. (I don't think this is a good thing, just stating the most realistic outcome based on extending what I've seen)
- chii 3y ago
  > decide not to worry about what previous generations considered core knowledge these sorts of things have already been happening for centuries. What was considered core knowledge depends on the environment one lives. Way back in prehistoric days, foraging and identifying edible plants would be considered core knowledge. How many people do this today? Prior to industrialization, most farmers will have some experience doing blacksmithing (for their own tools/repairs etc). That would be considered core knowledge that is lost today because industrialization made it unnecessary completely. Ditto with sewing, and many other skillsets that would be considered artisanal today. If knowledge acquisition turns out to be like that too, then it's fine imho.
thoughtlede 3y ago
LLMs are both language processing engines and knowledge bases. This article explores the knowledge base aspect of LLM and sheds light on the potential danger. The authors are well-justified in doing so because ChatGPT as a knowledge-bot is being used by many end users for its knowledge. However, to my knowledge, many enterprise applications that are being built using LLMs feed task-specific curated knowledge to LLMs. This mode of LLM use is encouraging. I do not think this article acknowledged this aspect of LLM use.
antisthenes 3y ago
This just means that in-person critical thinking skills will be at an even higher premium than ever. If knowledge collapse becomes evident, we'll dial back the use if AI, and a lot of "prompt monkey" businesses will go bankrupt.
- klyrs 3y ago
  > we'll dial back the use if AI Who, and how? This sounds suspiciously like the invisible hand
  - chii 3y ago
    If using the AI does not economically complete well with those who don't use said AI, then yes it will happen, and i attribute this to the invisible hand of darwinian competition.
    - klyrs 3y ago
      Yeah, that worked really well with cigarettes. People who used em were economically competitive with those who didn't.
    - justanotherjoe 3y ago
      I wonder how long natural providences like that hold as we get more and more alien to nature. We already learned we can f-up with global warming. We can fall off the cliff. It's not impossible. Maybe in the olden days when our boldest ideas were still relatively 'natural', we couldn't.
HarHarVeryFunny 3y ago
Maybe the problem (which seems easily fixable), is more "rizz collapse", aka blandness, than this "knowledge collapse". The model hasn't forgotten the diversity of material it was trained on, but outside of a context predicting a "long tail" response, it's going to predict a mid response. You can always prompt it to respond differently though. Blandness is more of an issue since that's what most-probable word-by-word generation is going to give you, rather than the less predictable, but more interesting, responses that an individual might give. Prompting could help by asking the model to reply in the idiosyncratic style of some celebrity, but this is likely to come across as a cheesy impression. Maybe the models could be trained to generate conditioned on a provided style sample, which could be long enough to avoid the cheesiness.
- thoughtlede 3y ago
  That's interesting. In keyword-based indexing solutions, a document vector is created using "term frequency inverse document frequency" scores. The idea is to pump up the document on the dimension where the document is unique compared to the other documents in the corpus. So when a query is issued with emphasis on a certain dimension, only documents that has higher scores in that dimension are returned. But the uniqueness in those solutions is based on keywords being used in the document, not concepts. What we need here to eliminate "blandness" is conceptual uniqueness. Maybe TF-IDF is still relevant to get there. Something to think about.
- jacobr1 3y ago
  Or introduce more noise or seeding to get more interesting responses. The `temperature` settings don't really satisfy this right now. I would like some determinism - but seeded randomly - so I can get similar responses if I like what is produced. Likewise some kind of metadata or explicability that allowed us to take a known style or featurespace of the model, perhaps from hand prompting, and then reuse-it with some degree is modification and maybe even combination from others would be very helpful. The work around adding model weights from fine-tuned seems directionally what I'm talking about, though that isn't the form I'd want to expose to users.
- thesz 3y ago
  > You can always prompt it to respond differently though. No, you cannot. I rarely do anything with LMs [1], but if I do I ask them to code "blocked clause decomposition" [2] in Haskell. Blocked clause decomposition is very, very simple as far as algorithm's implementation complexity goes and Haskell is just an unusual but simple implementation language. [1] https://arxiv.org/pdf/2403.07183.pdf [2] https://www.cs.utexas.edu/~marijn/publications/bcd.pdf I tried that on one less known model (my son sent me a link) and on Phind two times. The Phind's responces were very illuminating ones. On a first try, it tried to make me code that myself by pointing to a paper that does not contain any blocked clause decomposition algorithm(s), just worthless combinations of them and references to other papers. Of course, I read that paper before. When I specifically asked to provide me with code, it did not even provided me with the correct type of blocked clause, which should be a pair of a CNF clause and a literal that blocks it. On a second try I could not even pass through bullshitting phase - I just was not able to persuade a LM to write any code. The second try I attempted after I saw an advert here on HN about Phind's new and updated LM, better at coding. Try that youself with your unusual but known problem of choice. And this is exactly a problem of collapse of knowledge: you can't ask an LM to code you a problem it hasn't seen several dozen of times during training. You just cannot, you will not get a satisfying answer.
blurbleblurble 3y ago
Is it the AI that's the trouble or the hostile new information environment we're expected to navigate and survive? Expecting us to remain sane amidst these torrents of information without new tools for querying and filtering it is cruel.
- epgui 3y ago
  It can be (and is) both at the same time.
- deleted 3y ago
  [deleted]
- the_af 3y ago
  I only skimmed the paper, but judging by the abstract/intro and the definitions at the end, it's not concerned with "hostile" anything, but rather, with AI mediating access to knowledge. Specifically, they mention that "knowledge collapse" is the narrowing of the "human working knowledge" and the universe of things "worth knowing" or "epistemic limit", which may result in obscuring/hiding what they call "the long tail of knowledge" (they define all these terms). They explain this better than me, but I think it's all about constraining the human relation to knowledge by these new "recursive AI" processes, without any malice involved. I don't know if the paper mentions this, but I've thought about this (in less smart terms). It's probably not that AI is a radically new process in this sense but, like a lot of tech, it makes it faster. We often say "well, things were like this with [written language|printing press|the internet]" but isn't it possible that a relatively harmless process could be accelerated to a harmful level by technology, even if it's not "fundamentally" something new? Can we always fall back to the belief that "people from previous generations complained of $THING and it turned out ok, therefore $NEW_THING is probably OK too"?
  - OgsyedIE 3y ago
    The baseline will always shift to accommodate the conditions, it's how the hedonic treadmill evolved. The perspective we have, stuck inside the system, doesn't make the people of Denmark and the people of Burundi equally happy.
mark_l_watson 3y ago
I don’t disagree much with the paper’s premise, except that I am turned off whenever there is a handy new tool for us to use and then some people come up with reasons why using the new tool is a harmful thing to do. We humans evolved to be very flexible and naturally use what is in front of us to solve whatever problems we need to solve. For some tasks LLMs are very useful and for other tasks they don’t offer much help. I feel sorry for people who don’t learn how to use common tools because they are stuck in place or have unreasonable fears. It is reasonable to worry about how powerful actors might use AI for surveillance and removing privacy rights, but I strongly disagree that it is a bad thing for individuals to sometimes save time and sometimes get better results by manually using tools like Perplexity and ChatGPT, and automating things using LLMs as sources of general common sense knowledge, as language processors, data transformation, etc.
- readyman 3y ago
  >We humans evolved to be very flexible and naturally use what is in front of us to solve whatever problems we need to solve. Ha, yeah right. Almost all of us spent the vast majority of our time and energy just making dishonest rich people richer. In a world so full of exploitation, it should be no surprise that so many people are distrustful.
- thesz 3y ago
  The reliance on the tool in front of you can constrain you. Some tools are fads, some make you less of a thinking person. When I faced a second wave of IDEs I figured that any IDE is a fad - there will be better IDE in near future and your boss will require you to use it. The use of IDE can make you know less about project at hand, because you always can investigate, as opposed to internalize/remember. The difference? If you know internal relations, you can judge changes easier. You are better programmer that way and I would say a better person. The same goes to LMs - they help you not to know/remember and/or create thoughtfully. LMs constrain you to not know fully.
  - williamcotton 3y ago
    And the written word helps you to not remember anything, to paraphrase Socrates, so better get to memorizing the Odyssey so you can be an even better person.
627467 3y ago
> While large language models are trained on vast amounts of diverse data, they naturally generate output towards the 'center' of the distribution Yeah... Current gen AI does seem impressive but can't it be said that we have been preparing ourselves to be easily impressed by these "normal" results? The world is more homogenized than ever making it much easier to a bland intelligence to sound intelligent
dqft 3y ago
[dead]
ziofill 3y ago
This would be a problem if knowledge were a closed system. In that case it would suffice to invoke the data processing inequality and sooner or later your knowledge is all gone and you’re left with just noise. However, if AI enables new knowledge to be discovered then maybe there’s no problem in the end?
pella 3y ago
related github repo : - https://github.com/aristotle-tek/knowledge-collapse https://github.com/aristotle-tek/knowledge-collapse "replication code for "AI and the Problem of Knowledge Collapse"
visarga 3y ago
I don't think this paper goes deep enough, they don't consider for example structured diversity seeding for LLMs, which entails prompting the model to combine diverse entities, knowledge and skills selected at random from a list. Random combinations of conditioning context can lead the model to hyper-diversity. The Skill-Mix paper shows how this can work. https://arxiv.org/abs/2310.17567 https://arxiv.org/abs/2310.17567 LLMs have been trained on everything, when they are also prompted by diverse people with their own tasks and styles it won't lead to knowledge collapse. Two diverse worlds meeting each other in chat LLM interactions - the training corpus and people. But if you remove the human and just use fixed prompts, and then also narrow the training set with various methods, it might collapse.
- recursivesive 3y ago
  That's a good specific example/method, but the model in the paper specifically looks at when individuals will put effort into seeking this kind of diversity.
mediumsmart 3y ago
AI is the DaVinci enabler in a complex world
EVa5I7bHFq9mnYK 3y ago
AI is a clear negative from where I stand. As of last week, there are two new AI enabled cameras in my building, that are connected directly to the police. One of my banks recently started using (a completely useless) OpenAI chatgpt, while lying to me that I am talking to a live agent. When Googling something, I now skip the first page of results altogether, because it's all unendings walls of AI generated garbage. I can't read Twitter, Reddit and many other sites now, because they want to sell the content to AI training companies themselves. Up to 20% of my time on the web is now spent solving captchas, that are used to 1) help train AI and 2) to protect from AI scrapers. AI sends all my useful email to junk folder, so I now have to check two folders instead of one. Creative people are losing jobs right and left because AI can produce shitty "art" much cheaper. And this shitty "art" is now everywhere, polluting my brain. AI uses enormous amount of energy and contributes to climate catastrophe. AI is actively used to kill people in both currently ongoing wars. Today's Guardian: China will use AI to disrupt elections in the US, South Korea and India, Microsoft warns.
vouaobrasil 3y ago
> I don’t disagree much with the paper’s premise, except that I am turned off whenever there is a handy new tool for us to use and then some people come up with reasons why using the new tool is a harmful thing to do. I get turned off when people invent new tools.
osigurdson 3y ago
It sounds like the premise is, we will have fewer specialists and more dabblers. I think it could be true. LLMs are definitely a learning hack as never before could you ask an expert repeated dumb questions and a domain you know nothing about. Eventually you will learn something but it is quite different from being a traditional student and being guided by an expert - learning things that you didn't think to ask questions about or initially were not interested in (but later turn out to be important).

189 points by kmdupree 3y ago

So by this definition, do we already have "knowledge collapse" by Wikipedia? Because if you search for a random concept, that's usually the first hit, and it's also what countless other sources draw on.

rwbt 3y ago
Yes, we do kind of.
- hprotagonist 3y ago
  and i distinctly remember this critique being made at wikipedia's advent, as well. and it is not without justification! https://undark.org/2021/08/12/wikipedia-has-a-language-problem-heres-how-to-fix-it/ https://undark.org/2021/08/12/wikipedia-has-a-language-probl...
nickpsecurity 3y ago
The same warning was given for Google. Except those people added that it would reduce problem solving ability, too. People would get used to whatever simple, instant content rose to the top. They'd gradually lose some or all of their ability to figure out the same things on their own. One submission here was a tech guy at a school saying that was already happening where he worked.
- aoanla 3y ago
  I mean, it does - people search stuff all the time now, rather than thinking about it.
  - 082349872349872 3y ago
    IIRC, that was Socrates' complaint to Phaedrus about writing: that reading (because it was "high tech" at the time?) led only to an illusion of understanding. Elsewhere Phaedrus echoes with a very modern complaint (even though search engines wouldn't arrive for another 2'300 years): They would say in reply that he is a madman or a pedant who fancies that he is a physician because he has read something in a book, or has stumbled on a prescription or two, although he has no real understanding of the art of medicine. https://www.gutenberg.org/files/1636/1636-h/1636-h.htm https://www.gutenberg.org/files/1636/1636-h/1636-h.htm
    - jprete 3y ago
      Socrates wasn't wrong. Reading a lot gives you a partial understanding but it isn't complete without experiencing the thing for yourself. Arguably the Internet is the natural home of authoritatively stated but uninformed opinions - the exact result of reading a lot about a subject without having any experience of it.
      - contravariant 3y ago
        Ironically we can only know he was right by making the exact mistake he was warning us against.
    - voidhorse 3y ago
      I think the Phaedrus is all about the importance of practice. I've been reading a lot of math books lately, but I don't actually grok anything well until I sit down and try to reason through the material myself, write my own little proofs, try to deconstruct what's being said actively with pen and paper. Similarly, I understand a work of literature far more deeply if I take active notes, and/or write a small essay about my interpretation. I become a better writer by reading good writers and emulating them in my own writing practice. Writing was a threat to poets when the goal was still to recite a compelling live performance, which, to do this well, would require memorization and practice—today, still, we ask that actors do not have paper scripts in front of them when performing in a film. This is kind of the threat that tools like LLM's pose. Their power to generate decent results means that far more people will eschew practice for "good enough" LLM produced results. Creation will become even more transactional, and (many) people will quickly fail to "see the point" in practicing until we have a culture that's degraded even further than it already has today.
AlienRobot 3y ago
Personally, I think the problem is that people abuse Google for things it's really not designed to do, and they don't even realize that. Google is great at finding official webpages by their exact title. If you type the title of a news headline from the 90s, Google will give you the link to it. I think that is amazing. Basically anything that has a canonical URL, Google is good at finding. But when you search for "how to do X" for example, there will be several results that are perfectly valid and they will still need to be ranked in a list. Because it's not a "list" of results, it's ranking of relevancy. So to avoid showing spam, Google will push to the top websites it finds trustworthy. And now every top result comes from the same website. If you need an explanation for the Xz incident, for example, there is no canonical URL for it. There will be several news websites, youtube channels, etc. that have talked about it, competing to be the top result. Google still has to rank them even though the algorithm can't tell fact apart from parody, so no matter what Google does, Google will be the one judging which content most people will read when they want to know about a certain topic. To borrow my fellow robot's words, people are finding knowledge through an algorithmically curated aperture: Google's SERP. If they're evil, they have the power to control everyone on Earth. If they're good, they must be going insane with what to do with their users' crippling dependency on them as a source of truth.
- 082349872349872 3y ago
  I find Google isn't so good anymore for finding things by title; rather than being a search engine they are slowly becoming more like a politician answering reporters' questions, in that instead of returning results based on the terms I asked for, they insist on returning results for the terms they believe I should have asked for.
  - AlienRobot 3y ago
    Can you give me examples? Personally, I don't think I've ever had a problem finding things by their name or title on Google. If I type python, I get python's website, if I type hacker news, I get this website. I can type the name of an old program and get their webpage on sourceforge for example. I've also tested that it works with finding the official websites of celebrities, e.g. if you type eminem, you get his official website. It does seem to have biases toward recency and product pages, but finding things by their title isn't a thing Google is bad at.
the_af 3y ago
Probably Wikipedia is an early precursor of this problem, yes. It may not be entirely new, but AI might accelerate it to really harmful speeds. Just pondering this, of course.

JieJie 3y ago

The discussion section is quite illuminating. "While much recent attention has been on the problem of LLMs misleadingly presenting fiction as fact (hallucination), this may be less of an issue than the problem of representativeness across a distribution of possible responses. Hallucination of verifiable, concrete facts is often easy to correct for. Yet many real world questions do not have well-defined, verifiably true and false answers. If a user asks, for example, “What causes inflation?” and a LLM answers “monetary policy”, the problem isn’t one of hallucination, but of the failure to reflect the full-distribution of possible answers to the question, or at least provide an overview of the main schools of economic thought."

ben_w 3y ago
First thought: Oh no, they want LLMs to be even more vocal about nuance Second thought: People aren't going to read nuance Third thought: They should Fourth thought: Have you met people? They'll get angry with you for even suggesting it
voidhorse 3y ago
Right. It all hooks up to a naive, popular philosophy of knowledge that I think has been burgeoning since the birth of search engines: people today tend to think that everything is decomposable into atomic facts and that all questions have concrete, singular, atomic answers. Obviously, this could not be further from the truth, but I think ever since the advent of things like "answer cards" in search, people have been led into this technological bias trap of thinking that whatever answer the engine spits out must be the canonical answer and forget that we still have a broad range of questions and research areas we don't have anything close to definitive answers on. The problem is only exacerbated when people make this mistake in domains that will never have singular answers like politics and ethics.

knowsuchagency 3y ago

I feel like this has always been the case. The entire information economy is based on a few key publishers and figures. You see it in news, academia, social media -- there's orthodoxy everywhere. Not sure how AI is any different.

iraqmtpizza 3y ago
In the 1990s people read their town's newspaper. Now people in Arizona read the Daily Mail
- 48864w6ui 3y ago
  In the 1990s people who wanted to advertise in that town had to do so in local media. Now they can ad tech and the Daily Mail will arrange for it to be served.
- simonw 3y ago
  I worked for a local newspaper in Kansas around 2003/2004 and one thing I found surprising was that journalists there were frequently on the hook for writing up national stories - things that would come in off the wire services and then be re-written for the local audience.
  - iraqmtpizza 3y ago
    Did they write it like What in tarnation is going on in the Big Apple?
the_af 3y ago
It's probably different in that the few key publishers reached few people, but now AI is "sold" as for everyone and it's not unreasonable to expect it to become consumerized into a magic box of information, news and problem solving in the near future. It may not be a fundamentally new process, but what if it becomes so ubiquitous and fast that it becomes really harmful? And would there be a point of no return for society if this happened?
hackerlight 3y ago

karaterobot 3y ago

> Informally, we define knowledge collapse as the progressive narrowing over time (or over technological representations) of the set of information available to humans, along with a concomitant narrowing in the perceived availability and utility of different sets of information. > The main focus of the model is whether individuals decide to invest in innovation or learning ... in the ‘traditional’ way, through a possibly cheaper AI-enabled process, or not at all. The idea is to capture, for example, the difference between someone who does extensive research in an archive rather than just relying on readily-available materials, or someone who takes the time to read a full book rather than reading a two-paragraph LLM-generated summary. > Under these conditions, excessive reliance on AI-generated content over time leads to a curtailing of the eccentric and rare viewpoints that maintain a comprehensive vision of the world. My intuition is that AI will just accelerate the trends that the internet brought on, which is that eccentric viewpoints are actually pretty common, even ones based on research and in fact. The internet people mostly use has become relatively generic, consumed through a pretty narrow, curated aperture (social media). This feels analogous to getting it through AI, as described in the article. Yet, people are still learning about eccentric, marginal stuff all the time, especially compared to, say, 50 years ago. Assuming the AI's responses aren't artificially limited, people who are interested enough to look will still get to learn about topics in the long tail of the distribution, even in a world of ubiquitous AI. And they'll be able to dive as deeply into them as they do today. I'm not really worried about that. If anything, the knowledge collapse will be at the center. Basic liberal education topics are what will go away. Or rather, they will be offloaded to AI. In the same way that people say they don't need to learn arithmetic because they have a calculator, my guess is people will be more likely to decide not to worry about what previous generations considered core knowledge: history, geography, the canon, and so on. "I don't have to know it, I can look it up". That'll all go away even faster than it's going now. (I don't think this is a good thing, just stating the most realistic outcome based on extending what I've seen)

chii 3y ago
> decide not to worry about what previous generations considered core knowledge these sorts of things have already been happening for centuries. What was considered core knowledge depends on the environment one lives. Way back in prehistoric days, foraging and identifying edible plants would be considered core knowledge. How many people do this today? Prior to industrialization, most farmers will have some experience doing blacksmithing (for their own tools/repairs etc). That would be considered core knowledge that is lost today because industrialization made it unnecessary completely. Ditto with sewing, and many other skillsets that would be considered artisanal today. If knowledge acquisition turns out to be like that too, then it's fine imho.

thoughtlede 3y ago

LLMs are both language processing engines and knowledge bases. This article explores the knowledge base aspect of LLM and sheds light on the potential danger. The authors are well-justified in doing so because ChatGPT as a knowledge-bot is being used by many end users for its knowledge. However, to my knowledge, many enterprise applications that are being built using LLMs feed task-specific curated knowledge to LLMs. This mode of LLM use is encouraging. I do not think this article acknowledged this aspect of LLM use.

antisthenes 3y ago

This just means that in-person critical thinking skills will be at an even higher premium than ever. If knowledge collapse becomes evident, we'll dial back the use if AI, and a lot of "prompt monkey" businesses will go bankrupt.

klyrs 3y ago
> we'll dial back the use if AI Who, and how? This sounds suspiciously like the invisible hand
- chii 3y ago
  If using the AI does not economically complete well with those who don't use said AI, then yes it will happen, and i attribute this to the invisible hand of darwinian competition.
  - klyrs 3y ago
    Yeah, that worked really well with cigarettes. People who used em were economically competitive with those who didn't.
  - justanotherjoe 3y ago
    I wonder how long natural providences like that hold as we get more and more alien to nature. We already learned we can f-up with global warming. We can fall off the cliff. It's not impossible. Maybe in the olden days when our boldest ideas were still relatively 'natural', we couldn't.

HarHarVeryFunny 3y ago

Maybe the problem (which seems easily fixable), is more "rizz collapse", aka blandness, than this "knowledge collapse". The model hasn't forgotten the diversity of material it was trained on, but outside of a context predicting a "long tail" response, it's going to predict a mid response. You can always prompt it to respond differently though. Blandness is more of an issue since that's what most-probable word-by-word generation is going to give you, rather than the less predictable, but more interesting, responses that an individual might give. Prompting could help by asking the model to reply in the idiosyncratic style of some celebrity, but this is likely to come across as a cheesy impression. Maybe the models could be trained to generate conditioned on a provided style sample, which could be long enough to avoid the cheesiness.

thoughtlede 3y ago
That's interesting. In keyword-based indexing solutions, a document vector is created using "term frequency inverse document frequency" scores. The idea is to pump up the document on the dimension where the document is unique compared to the other documents in the corpus. So when a query is issued with emphasis on a certain dimension, only documents that has higher scores in that dimension are returned. But the uniqueness in those solutions is based on keywords being used in the document, not concepts. What we need here to eliminate "blandness" is conceptual uniqueness. Maybe TF-IDF is still relevant to get there. Something to think about.
jacobr1 3y ago
Or introduce more noise or seeding to get more interesting responses. The `temperature` settings don't really satisfy this right now. I would like some determinism - but seeded randomly - so I can get similar responses if I like what is produced. Likewise some kind of metadata or explicability that allowed us to take a known style or featurespace of the model, perhaps from hand prompting, and then reuse-it with some degree is modification and maybe even combination from others would be very helpful. The work around adding model weights from fine-tuned seems directionally what I'm talking about, though that isn't the form I'd want to expose to users.
thesz 3y ago
> You can always prompt it to respond differently though. No, you cannot. I rarely do anything with LMs [1], but if I do I ask them to code "blocked clause decomposition" [2] in Haskell. Blocked clause decomposition is very, very simple as far as algorithm's implementation complexity goes and Haskell is just an unusual but simple implementation language. [1] https://arxiv.org/pdf/2403.07183.pdf [2] https://www.cs.utexas.edu/~marijn/publications/bcd.pdf I tried that on one less known model (my son sent me a link) and on Phind two times. The Phind's responces were very illuminating ones. On a first try, it tried to make me code that myself by pointing to a paper that does not contain any blocked clause decomposition algorithm(s), just worthless combinations of them and references to other papers. Of course, I read that paper before. When I specifically asked to provide me with code, it did not even provided me with the correct type of blocked clause, which should be a pair of a CNF clause and a literal that blocks it. On a second try I could not even pass through bullshitting phase - I just was not able to persuade a LM to write any code. The second try I attempted after I saw an advert here on HN about Phind's new and updated LM, better at coding. Try that youself with your unusual but known problem of choice. And this is exactly a problem of collapse of knowledge: you can't ask an LM to code you a problem it hasn't seen several dozen of times during training. You just cannot, you will not get a satisfying answer.

blurbleblurble 3y ago

Is it the AI that's the trouble or the hostile new information environment we're expected to navigate and survive? Expecting us to remain sane amidst these torrents of information without new tools for querying and filtering it is cruel.

epgui 3y ago
It can be (and is) both at the same time.
deleted 3y ago
[deleted]
the_af 3y ago
I only skimmed the paper, but judging by the abstract/intro and the definitions at the end, it's not concerned with "hostile" anything, but rather, with AI mediating access to knowledge. Specifically, they mention that "knowledge collapse" is the narrowing of the "human working knowledge" and the universe of things "worth knowing" or "epistemic limit", which may result in obscuring/hiding what they call "the long tail of knowledge" (they define all these terms). They explain this better than me, but I think it's all about constraining the human relation to knowledge by these new "recursive AI" processes, without any malice involved. I don't know if the paper mentions this, but I've thought about this (in less smart terms). It's probably not that AI is a radically new process in this sense but, like a lot of tech, it makes it faster. We often say "well, things were like this with [written language|printing press|the internet]" but isn't it possible that a relatively harmless process could be accelerated to a harmful level by technology, even if it's not "fundamentally" something new? Can we always fall back to the belief that "people from previous generations complained of $THING and it turned out ok, therefore $NEW_THING is probably OK too"?
- OgsyedIE 3y ago
  The baseline will always shift to accommodate the conditions, it's how the hedonic treadmill evolved. The perspective we have, stuck inside the system, doesn't make the people of Denmark and the people of Burundi equally happy.

mark_l_watson 3y ago

I don’t disagree much with the paper’s premise, except that I am turned off whenever there is a handy new tool for us to use and then some people come up with reasons why using the new tool is a harmful thing to do. We humans evolved to be very flexible and naturally use what is in front of us to solve whatever problems we need to solve. For some tasks LLMs are very useful and for other tasks they don’t offer much help. I feel sorry for people who don’t learn how to use common tools because they are stuck in place or have unreasonable fears. It is reasonable to worry about how powerful actors might use AI for surveillance and removing privacy rights, but I strongly disagree that it is a bad thing for individuals to sometimes save time and sometimes get better results by manually using tools like Perplexity and ChatGPT, and automating things using LLMs as sources of general common sense knowledge, as language processors, data transformation, etc.

readyman 3y ago
>We humans evolved to be very flexible and naturally use what is in front of us to solve whatever problems we need to solve. Ha, yeah right. Almost all of us spent the vast majority of our time and energy just making dishonest rich people richer. In a world so full of exploitation, it should be no surprise that so many people are distrustful.
thesz 3y ago
The reliance on the tool in front of you can constrain you. Some tools are fads, some make you less of a thinking person. When I faced a second wave of IDEs I figured that any IDE is a fad - there will be better IDE in near future and your boss will require you to use it. The use of IDE can make you know less about project at hand, because you always can investigate, as opposed to internalize/remember. The difference? If you know internal relations, you can judge changes easier. You are better programmer that way and I would say a better person. The same goes to LMs - they help you not to know/remember and/or create thoughtfully. LMs constrain you to not know fully.
- williamcotton 3y ago
  And the written word helps you to not remember anything, to paraphrase Socrates, so better get to memorizing the Odyssey so you can be an even better person.

627467 3y ago

> While large language models are trained on vast amounts of diverse data, they naturally generate output towards the 'center' of the distribution Yeah... Current gen AI does seem impressive but can't it be said that we have been preparing ourselves to be easily impressed by these "normal" results? The world is more homogenized than ever making it much easier to a bland intelligence to sound intelligent

dqft 3y ago

[dead]

ziofill 3y ago

This would be a problem if knowledge were a closed system. In that case it would suffice to invoke the data processing inequality and sooner or later your knowledge is all gone and you’re left with just noise. However, if AI enables new knowledge to be discovered then maybe there’s no problem in the end?

pella 3y ago

related github repo : - https://github.com/aristotle-tek/knowledge-collapse https://github.com/aristotle-tek/knowledge-collapse "replication code for "AI and the Problem of Knowledge Collapse"

visarga 3y ago

I don't think this paper goes deep enough, they don't consider for example structured diversity seeding for LLMs, which entails prompting the model to combine diverse entities, knowledge and skills selected at random from a list. Random combinations of conditioning context can lead the model to hyper-diversity. The Skill-Mix paper shows how this can work. https://arxiv.org/abs/2310.17567 https://arxiv.org/abs/2310.17567 LLMs have been trained on everything, when they are also prompted by diverse people with their own tasks and styles it won't lead to knowledge collapse. Two diverse worlds meeting each other in chat LLM interactions - the training corpus and people. But if you remove the human and just use fixed prompts, and then also narrow the training set with various methods, it might collapse.

recursivesive 3y ago
That's a good specific example/method, but the model in the paper specifically looks at when individuals will put effort into seeking this kind of diversity.

mediumsmart 3y ago

AI is the DaVinci enabler in a complex world

EVa5I7bHFq9mnYK 3y ago

AI is a clear negative from where I stand. As of last week, there are two new AI enabled cameras in my building, that are connected directly to the police. One of my banks recently started using (a completely useless) OpenAI chatgpt, while lying to me that I am talking to a live agent. When Googling something, I now skip the first page of results altogether, because it's all unendings walls of AI generated garbage. I can't read Twitter, Reddit and many other sites now, because they want to sell the content to AI training companies themselves. Up to 20% of my time on the web is now spent solving captchas, that are used to 1) help train AI and 2) to protect from AI scrapers. AI sends all my useful email to junk folder, so I now have to check two folders instead of one. Creative people are losing jobs right and left because AI can produce shitty "art" much cheaper. And this shitty "art" is now everywhere, polluting my brain. AI uses enormous amount of energy and contributes to climate catastrophe. AI is actively used to kill people in both currently ongoing wars. Today's Guardian: China will use AI to disrupt elections in the US, South Korea and India, Microsoft warns.

vouaobrasil 3y ago

> I don’t disagree much with the paper’s premise, except that I am turned off whenever there is a handy new tool for us to use and then some people come up with reasons why using the new tool is a harmful thing to do. I get turned off when people invent new tools.

osigurdson 3y ago

It sounds like the premise is, we will have fewer specialists and more dabblers. I think it could be true. LLMs are definitely a learning hack as never before could you ask an expert repeated dumb questions and a domain you know nothing about. Eventually you will learn something but it is quite different from being a traditional student and being guided by an expert - learning things that you didn't think to ask questions about or initially were not interested in (but later turn out to be important).