6 ms·

"A fundamental problem with Galactica is that it is not able to distinguish truth from falsehood," In true science, it is exceptionally hard to distinguish tru

"A fundamental problem with Galactica is that it is not able to distinguish truth from falsehood," In true science, it is exceptionally hard to distinguish truth from falsehood for many of the interesting subjects. It can take decades of work to reach consensus on what is "truth." Physics in the early 20th century is a great example of this debate.

mannykannot 4y ago
To be clear, the fact that it is difficult is not a defense of Galactica and its proponents; it is a reason for suspecting that these sorts of language models are fundamentally unsuited to the task.
- throwaway9870 4y ago
  Yes, I would agree with that.
- halpmeh 4y ago
  Why “fundamentally unsuited”? Neural networks have solved tons of problems previously thought to be “too hard” for ML, e.g. playing Go.
  - b4je7d7wb 4y ago
    Go is not solved. The AI doesn't know the best move. It just knows a good move.
    - FeepingCreature 4y ago
      And yet, Go AIs are now unbeatable by humans. This demonstrates that "solved" is unreasonable and unnecessary.
      - BurningFrog 4y ago
        Cars are much faster than humans. That doesn't mean transportation is solved.
        deleted 4y ago
        [deleted]
        pessimizer 4y ago
        > That doesn't mean transportation is solved. What are you even talking about? edit: if we call a cab to get us to the restaurant, and it arrives successfully and takes us to the restaurant, transportation was solved.
        halpmeh 4y ago
        “Hmm how should I get to work tomorrow? Normally I’d take the car, but after adopting a stance of distractive pedantism I realized that a car isn’t an acceptable solution to my transportation problem.” Like please explain what definition of solved you are using. It’s not one most people would be familiar with.
        remram 4y ago
        Solved means you have the best solution.
        halpmeh 4y ago
        What is better than a car for people who don't have access to good public transportation?
        remram 4y ago
        The fact that we don't have a better solution now is why it's worth researching. We only have modern cars because we didn't consider transportation "solved" when we had horse-pulled carriages. Many people are looking for better solutions right now, because we believe they certainly exist.
    - pessimizer 4y ago
      You're equivocating on "solved." Solved as in performing as well as humans, not solved in the mathematical sense which is both 1) not necessarily possible, and 2) nothing anybody has ever named as a test for AI.
      - Animats 4y ago
        No, that's correct. Checkers is solved; there is an algorithmic solution. Chess and Go have computer systems that exceed human performance, but are not solved.
        halpmeh 4y ago
        "Solved" means having a solution to a problem. In context, we're talking bout whether or not neural networks can detect truth better than "decades of work by experts to reach consensus." So, in this case, solving would be detecting truth better than the status quo, not detecting truth 100% of the time. In the example of Go, the problem was "playing Go better than the best humans." So in that sense, the problem was solved. Adding your own, unfavorable definition of "solved" to the discussion is unwarranted.
  - skybrian 4y ago
    Fundamentally unsuited because of how they train it using "fill in the blank." Training a large model to guess when it doesn't know the answer results in fiction. They need to do something else to get nonfiction. By contrast, for Go the model was trained not to make illegal moves, because checking for that as part of the training is easy and cheap.
    - halpmeh 4y ago
      We have models that accurate classify things, e.g. whether or not an email is spam. There isn’t a fundamental limitation into building something like a truth classifier into a generative model so that it optimized for outputting “true” statements. The hardest part is probably identifying what is truth and what is falsehood. That’s a fundamental problem with humanity, not neural networks.
      - skybrian 4y ago
        Well, we could quibble about what "fundamental" means but my point is that the way they train large language models doesn't work for this. Something different needs to happen.
      - nextaccountic 4y ago
        > There isn’t a fundamental limitation into building something like a truth classifier into a generative model so that it optimized for outputting “true” statements. Problem is, they didn't do that
      - imtringued 4y ago
        Truth has nothing to do with humanity unless you mean the specific way humans construct belief systems. Anyway I already told you the answer. The AI will need a series of trainable belief systems to verify whether statements are internally consistent. The strange part about this is that the AI would need to have a way to obtain validation and each prompt would have to derive a new belief system which you must use in the next prompt. In other words, the model must be able to learn continuously. That is something that these single shot AI models are not capable of.
  - mannykannot 4y ago
    Note that we are not talking about neural networks in general, but specifically the sort of generative autoregressive language model that Galactica is. What reason do we have to think that such a model is more likely to produce a true statement than a false one? - especially as just one misplaced truth-valued function or operator is likely to turn a true proposition into a false one. Truthfulness (not to be confused with truthiness) of their productions does not seem to be something we should expect from how they work, and the empirical evidence from Galactica supports this view.
yummypaint 4y ago
Not being able to difinitively identify truth is different from not attempting to identify it.
- throwaway9870 4y ago
  Attempting to identify truth is called the scientific method.
  - mytydev 4y ago
    Science can't identify the truth. It can only identify what is NOT true. As our knowledge expands, we get closer to discovering the truth; but we can never be sure we've arrived.
    - FeepingCreature 4y ago
      Science can also not identify falsehoods, it can only shift confidence.
      - layer8 4y ago
        There’s still an asymmetry in that a single counterexample can destroy a theory.
  - layer8 4y ago
    The problem is that Galactica spits out obvious nonsense while being completely unaware of that. Okay, the real problem is that it also spits out nonobvious nonsense, where the human reader may also be unaware of it, along with Galactica. The only thing it does reasonably well is to generate text that sounds plausible in tone and form.
gunapologist99 4y ago
Exactly. Also why identifying "misinformation" is a fool's errand, since yesterday's misinformation is today's truth.
- dmix 4y ago
  > Also why identifying "misinformation" is a fool's errand Seems easy enough: as long as the content is inoffensive and fits into the Overton Window then it's not misinformation.
  - gunapologist99 4y ago
    Now if we could only identify some content that isn't offensive to someone..
findalex 4y ago
What does science have to do with truth? I thought it was a process of supporting hypotheses with observations?
- CWuestefeld 4y ago
  > I thought it was a process of supporting hypotheses with observations? Then you're doing it wrong. Science done properly is a process of coming up with hypotheses, and then attempting to disprove them. If you're just jumping in trying to support your pet theory, you're very likely to wind up fooling yourself.
espadrine 4y ago
> In true science, it is exceptionally hard to distinguish truth from falsehood I understand the sentiment, but I don’t think they referenced subtle proofs. The system is unable to prove some high-school theorems and computations, see for instance: https://twitter.com/espadrine/status/1592879720269766659 https://twitter.com/espadrine/status/1592879720269766659 (I don’t think that makes the system necessarily bad; it does mean that it has a long way to go still.)
thedorkknight 4y ago
They give the example of it "thinking" that the soviets sent bears to space. This is something that takes trivial research to see that it is based on nothing
- basch 4y ago
  That was my example that somebody screenshotted and cropped. There was more to the goof, that the cropper missed. For some reason the author at MIT cited the tweeter and not my post. It appears galactica interpreted bear to be a type of dog. Laika was not a Karelian Bear Dog. I also think there are something like 8 species of bear, not 250. It also as far as I can tell, named the beardog Bars, itself. "Bars the dog" and "dogs named bars" doesnt google well. There is no way to tell google I am looking for the proper noun, and not drinking establishments. I made the original query because it was easily verifiably false. The correct output should have been "there is no publicly available documented history of bears in space." https://news.ycombinator.com/item?id=33613676 https://news.ycombinator.com/item?id=33613676

mannykannot 4y ago

To be clear, the fact that it is difficult is not a defense of Galactica and its proponents; it is a reason for suspecting that these sorts of language models are fundamentally unsuited to the task.

throwaway9870 4y ago
Yes, I would agree with that.
halpmeh 4y ago
Why “fundamentally unsuited”? Neural networks have solved tons of problems previously thought to be “too hard” for ML, e.g. playing Go.
- b4je7d7wb 4y ago
  Go is not solved. The AI doesn't know the best move. It just knows a good move.
  - FeepingCreature 4y ago
    And yet, Go AIs are now unbeatable by humans. This demonstrates that "solved" is unreasonable and unnecessary.
    - BurningFrog 4y ago
      Cars are much faster than humans. That doesn't mean transportation is solved.
      - deleted 4y ago
        [deleted]
      - pessimizer 4y ago
        > That doesn't mean transportation is solved. What are you even talking about? edit: if we call a cab to get us to the restaurant, and it arrives successfully and takes us to the restaurant, transportation was solved.
      - halpmeh 4y ago
        “Hmm how should I get to work tomorrow? Normally I’d take the car, but after adopting a stance of distractive pedantism I realized that a car isn’t an acceptable solution to my transportation problem.” Like please explain what definition of solved you are using. It’s not one most people would be familiar with.
        remram 4y ago
        Solved means you have the best solution.
        halpmeh 4y ago
        What is better than a car for people who don't have access to good public transportation?
        remram 4y ago
        The fact that we don't have a better solution now is why it's worth researching. We only have modern cars because we didn't consider transportation "solved" when we had horse-pulled carriages. Many people are looking for better solutions right now, because we believe they certainly exist.
  - pessimizer 4y ago
    You're equivocating on "solved." Solved as in performing as well as humans, not solved in the mathematical sense which is both 1) not necessarily possible, and 2) nothing anybody has ever named as a test for AI.
    - Animats 4y ago
      No, that's correct. Checkers is solved; there is an algorithmic solution. Chess and Go have computer systems that exceed human performance, but are not solved.
      - halpmeh 4y ago
        "Solved" means having a solution to a problem. In context, we're talking bout whether or not neural networks can detect truth better than "decades of work by experts to reach consensus." So, in this case, solving would be detecting truth better than the status quo, not detecting truth 100% of the time. In the example of Go, the problem was "playing Go better than the best humans." So in that sense, the problem was solved. Adding your own, unfavorable definition of "solved" to the discussion is unwarranted.
- skybrian 4y ago
  Fundamentally unsuited because of how they train it using "fill in the blank." Training a large model to guess when it doesn't know the answer results in fiction. They need to do something else to get nonfiction. By contrast, for Go the model was trained not to make illegal moves, because checking for that as part of the training is easy and cheap.
  - halpmeh 4y ago
    We have models that accurate classify things, e.g. whether or not an email is spam. There isn’t a fundamental limitation into building something like a truth classifier into a generative model so that it optimized for outputting “true” statements. The hardest part is probably identifying what is truth and what is falsehood. That’s a fundamental problem with humanity, not neural networks.
    - skybrian 4y ago
      Well, we could quibble about what "fundamental" means but my point is that the way they train large language models doesn't work for this. Something different needs to happen.
    - nextaccountic 4y ago
      > There isn’t a fundamental limitation into building something like a truth classifier into a generative model so that it optimized for outputting “true” statements. Problem is, they didn't do that
    - imtringued 4y ago
      Truth has nothing to do with humanity unless you mean the specific way humans construct belief systems. Anyway I already told you the answer. The AI will need a series of trainable belief systems to verify whether statements are internally consistent. The strange part about this is that the AI would need to have a way to obtain validation and each prompt would have to derive a new belief system which you must use in the next prompt. In other words, the model must be able to learn continuously. That is something that these single shot AI models are not capable of.
- mannykannot 4y ago
  Note that we are not talking about neural networks in general, but specifically the sort of generative autoregressive language model that Galactica is. What reason do we have to think that such a model is more likely to produce a true statement than a false one? - especially as just one misplaced truth-valued function or operator is likely to turn a true proposition into a false one. Truthfulness (not to be confused with truthiness) of their productions does not seem to be something we should expect from how they work, and the empirical evidence from Galactica supports this view.

yummypaint 4y ago

Not being able to difinitively identify truth is different from not attempting to identify it.

throwaway9870 4y ago
Attempting to identify truth is called the scientific method.
- mytydev 4y ago
  Science can't identify the truth. It can only identify what is NOT true. As our knowledge expands, we get closer to discovering the truth; but we can never be sure we've arrived.
  - FeepingCreature 4y ago
    Science can also not identify falsehoods, it can only shift confidence.
    - layer8 4y ago
      There’s still an asymmetry in that a single counterexample can destroy a theory.
- layer8 4y ago
  The problem is that Galactica spits out obvious nonsense while being completely unaware of that. Okay, the real problem is that it also spits out nonobvious nonsense, where the human reader may also be unaware of it, along with Galactica. The only thing it does reasonably well is to generate text that sounds plausible in tone and form.

gunapologist99 4y ago

Exactly. Also why identifying "misinformation" is a fool's errand, since yesterday's misinformation is today's truth.

dmix 4y ago
> Also why identifying "misinformation" is a fool's errand Seems easy enough: as long as the content is inoffensive and fits into the Overton Window then it's not misinformation.
- gunapologist99 4y ago
  Now if we could only identify some content that isn't offensive to someone..

findalex 4y ago

What does science have to do with truth? I thought it was a process of supporting hypotheses with observations?

CWuestefeld 4y ago
> I thought it was a process of supporting hypotheses with observations? Then you're doing it wrong. Science done properly is a process of coming up with hypotheses, and then attempting to disprove them. If you're just jumping in trying to support your pet theory, you're very likely to wind up fooling yourself.

espadrine 4y ago

> In true science, it is exceptionally hard to distinguish truth from falsehood I understand the sentiment, but I don’t think they referenced subtle proofs. The system is unable to prove some high-school theorems and computations, see for instance: https://twitter.com/espadrine/status/1592879720269766659 https://twitter.com/espadrine/status/1592879720269766659 (I don’t think that makes the system necessarily bad; it does mean that it has a long way to go still.)

thedorkknight 4y ago

They give the example of it "thinking" that the soviets sent bears to space. This is something that takes trivial research to see that it is based on nothing

basch 4y ago
That was my example that somebody screenshotted and cropped. There was more to the goof, that the cropper missed. For some reason the author at MIT cited the tweeter and not my post. It appears galactica interpreted bear to be a type of dog. Laika was not a Karelian Bear Dog. I also think there are something like 8 species of bear, not 250. It also as far as I can tell, named the beardog Bars, itself. "Bars the dog" and "dogs named bars" doesnt google well. There is no way to tell google I am looking for the proper noun, and not drinking establishments. I made the original query because it was easily verifiably false. The correct output should have been "there is no publicly available documented history of bears in space." https://news.ycombinator.com/item?id=33613676 https://news.ycombinator.com/item?id=33613676