3 ms·

If you ask an LLM "did Alice on the X forum write Y?", you can get a justifiable answer. Sure, but that’s not a very interesting question and search engines te

If you ask an LLM "did Alice on the X forum write Y?", you can get a justifiable answer. Sure, but that’s not a very interesting question and search engines tend to be much better at answering it anyway. Historians have access to more than just texts about the past. They have access to physical evidence. Radiometric dating, artifacts, geographic data about where things were found. At the same time, if you ask historians about (say) the Peloponnesian War they will tell you about the account of Thucydides. What they won’t say is that they know about it personally. That’s a mistake a lot of non-scientists make about science. They’ll hear scientists say things like “we know this” or “we know that” and think that they are committing to some privileged knowledge of the world. They aren’t. They’re using the words “we know X” as shorthand for “our best theory about X.” In fact, many scientists identify as scientific anti-realists [1] which means they don’t believe science has any true facts about the nature of reality at all and that the only job of science is to make better predictions about the outcome of experiments. Of course, if you take anti-realism to its logical conclusion then you have to admit that fossils are only evidence that you may find more fossils in the future, and that you have no basis to talk about the existence of dinosaurs. Personally, I think our actual knowledge of things directly outside our personal experience is very limited. But then I want to encourage people not just to believe everything they read. People should be exploring and trying their own science experiments and trying to build things to understand how the world works. [1] https://plato.stanford.edu/entries/scientific-realism/#AntiFoilForScieReal https://plato.stanford.edu/entries/scientific-realism/#AntiF...

persnickety 3y ago
That a question is not interesting does not rule out the justifiability of the answer, which you seemed to assert. > Historians have access to more than just texts about the past. They have access to physical evidence. Radiometric dating, artifacts, geographic data about where things were found. Correct. That's still all indirect, "third-party" evidence. Does that make answers based on it impossible to justify? To make this analogy even closer to LLMs, let's take historian scholars (linguists? undergrads?) who have nothing to study but books. No experiments, no carbon dating, just texts. Is their entire knowledge about the past impossible to justify?
- chongli 3y ago
  Again, you're thinking in binaries. Justification is a spectrum. Sometimes you have no evidence, sometimes a little, sometimes a lot. Historians who only have books to go on have frequently said the accounts they're using should be treated as only that: accounts, not facts about what happened. For example, even a figure as popular as Socrates is "known" only through the accounts of Plato and Xenophon. What do we know about him? The two accounts are in conflict about many details of his personality. So we can attribute very little in the writings of Plato to actual opinions of Socrates. Scholars treat these books as the opinions of Plato, expressed through dialogues of Socrates.
  - persnickety 3y ago
    I'm questioning your binary. As far as I can tell, you're saying that LLMs cannot make justified statements because all evidence they have access to is indirect text. Therefore, you're stating that no amount of text can justify a belief. Am I misrepresenting your position? But when I present an analogous situation, except with historians playing the main role, you seem to accept the justifiability of their statements on a spectrum, rather than rejecting it based on the evidence being textual and indirect. This is the crux of the discussion, without explaining which we won't come to a mutual understanding (much less acceptance).
    - chongli 3y ago
      Historians who don’t have access to anything other than text (which is your hypothetical situation) cannot justify any claim about real world events in the past. However, this does not happen in real life. Text does not fall from the heavens into the minds of historians. It comes in books, manuscripts, stone tablets, etc. Unlike digital text (which is the only thing LLMs have access to), physical texts have physical properties which can be examined experimentally. They can be subject to radiometric dating, they typically are found in a particular place (which is usually recorded), and they have version and copyright information which enables them to be traced back to the original. In the case where the original manuscript is lost (such as Thucydides’ history of the Peloponnesian War), we have to rely on copied manuscripts by other scribes, often made centuries later. All of this is to say that it can be very difficult to trace a book back to its original author, especially without the original manuscript, but the evidence historians use to piece all this together is generally not in the text itself. It’s physical evidence. Exactly the thing that LLMs have no access to.

persnickety 3y ago

That a question is not interesting does not rule out the justifiability of the answer, which you seemed to assert. > Historians have access to more than just texts about the past. They have access to physical evidence. Radiometric dating, artifacts, geographic data about where things were found. Correct. That's still all indirect, "third-party" evidence. Does that make answers based on it impossible to justify? To make this analogy even closer to LLMs, let's take historian scholars (linguists? undergrads?) who have nothing to study but books. No experiments, no carbon dating, just texts. Is their entire knowledge about the past impossible to justify?

chongli 3y ago
Again, you're thinking in binaries. Justification is a spectrum. Sometimes you have no evidence, sometimes a little, sometimes a lot. Historians who only have books to go on have frequently said the accounts they're using should be treated as only that: accounts, not facts about what happened. For example, even a figure as popular as Socrates is "known" only through the accounts of Plato and Xenophon. What do we know about him? The two accounts are in conflict about many details of his personality. So we can attribute very little in the writings of Plato to actual opinions of Socrates. Scholars treat these books as the opinions of Plato, expressed through dialogues of Socrates.
- persnickety 3y ago
  I'm questioning your binary. As far as I can tell, you're saying that LLMs cannot make justified statements because all evidence they have access to is indirect text. Therefore, you're stating that no amount of text can justify a belief. Am I misrepresenting your position? But when I present an analogous situation, except with historians playing the main role, you seem to accept the justifiability of their statements on a spectrum, rather than rejecting it based on the evidence being textual and indirect. This is the crux of the discussion, without explaining which we won't come to a mutual understanding (much less acceptance).
  - chongli 3y ago
    Historians who don’t have access to anything other than text (which is your hypothetical situation) cannot justify any claim about real world events in the past. However, this does not happen in real life. Text does not fall from the heavens into the minds of historians. It comes in books, manuscripts, stone tablets, etc. Unlike digital text (which is the only thing LLMs have access to), physical texts have physical properties which can be examined experimentally. They can be subject to radiometric dating, they typically are found in a particular place (which is usually recorded), and they have version and copyright information which enables them to be traced back to the original. In the case where the original manuscript is lost (such as Thucydides’ history of the Peloponnesian War), we have to rely on copied manuscripts by other scribes, often made centuries later. All of this is to say that it can be very difficult to trace a book back to its original author, especially without the original manuscript, but the evidence historians use to piece all this together is generally not in the text itself. It’s physical evidence. Exactly the thing that LLMs have no access to.