4 ms·

Why is it a binary choice? "Most students would answer X, due to this common misconception about Y". As we have seen from history, there is not often an absolu

Why is it a binary choice? "Most students would answer X, due to this common misconception about Y". As we have seen from history, there is not often an absolute truth to questions, only clusters of truths. We want our LLM to be able to perform reasoning, mathematics, and science, but expecting absolute truths in anything outside of those fields is a bit much. Wikipedia often takes a good approach here and strikes this balance well. You can represent the "crazies" and show their reasoning, and then draw attention to how it is commonly refuted. You don't get this if you just omit the "crazies" to begin with. Edit: What I'm saying is that you can include/represent these adjacent clusters of answers instead of asserting one cluster to be true and allow the correct answer to be represented organically. And guess what? Some people will still disagree for whatever reason they have to disagree. We should be taking all this AI alignment money and putting it into education imo.

kristopolous 3y ago
That's not how bullshit works. An expertise is needed from the reader when presented with false material. That's why so many people think parody news articles are real. If you have ever gone through any comment thread on the internet about climate change you'll find a large volume of climate deniers citing God, greedy liberals, George Soros, whatever. The people working with these fictions can always just add more fiction to fill in whatever hole. They aren't bound by reality or credulity. Actually understanding global climate systems takes an incredible amount of study. You just can't blurt out a few paragraphs of words and get people to stop believing in Jewish weather machine conspiracies or whatever they're thinking. We would have been done with that by now if this were the case. Dislodging bullshit is hard. Humanity only started figuring out how a few hundred years ago and we're still fucking it up - such as in the replication crisis
- bick_nyers 3y ago
  Right but I feel like in your argument there is an implicit assumption that the only way to get ChatGPT etc. to understand say a global climate system is by omitting false information entirely. Why can't an LLM extract a signal from the noise? It seems as though it does that already. If you're worried that there is more text on the internet saying that the Earth is flat than there is text saying that the Earth is round, then I understand where you are coming from, but then how would ANYONE come to believe that the Earth is round on their own? By learning how to reason. Edit: My point is that if the LLM can't weed through bullshit properly then how can we expect it to get anything right at all? We can't just feed it a continuously updated list of facts that are true and facts that are false until the end of time.
  - eropple 3y ago
    > Right but I feel like in your argument there is an implicit assumption that the only way to get ChatGPT etc. to understand say a global climate system is by omitting false information entirely. Why can't an LLM extract a signal from the noise? It seems as though it does that already. Because it's not thinking, it's looking at probabilities. If you don't prune branches, you've created a disinformation radiator if that information is sufficiently populous within your corpus to be a pretty-good-bet. The best analogy I've come up with is "it's like if you had an objective reality based on Wikipedia edit wars."
    - kristopolous 3y ago
      Exactly. Especially when there's some hustle behind lying to people. Phony medical advice is the classic one here. People do it to sell their sugar pills or do some Peter Popoff style healing touch hustle. Some scams, like MLM, encourage massive volumes of this bullshit to be created and disseminated. The democracy of information does not lead to an increased accuracy based on whatever information spreads the most. The internet is a multi-decade long global proof of this. There's a reason that classrooms have 1 teacher and say 24 students - only 4% of that room is to be trusted with the material. To know if something is true, you need to eventually have some kind of external confirmation. Karl Popper wrote a lot about this. These chat systems can make predictions about the existing corpus but it can't like roll around town and do actual science to confirm things. If all truth must be derived from self-contained systems than you have to make sure your self-contained system isn't bullshit. A game for instance, can have its own rules and consistencies and things can be rationalized through the game. But how much that maps to a reality cannot be concluded strictly from inside the game. For instance, if you want expertise in masonry and construction you need to make sure you aren't just feeding your AI Tetris and SimCity.
    - bick_nyers 3y ago
      Just because it is a probability doesn't mean that it's strictly the probability derived from frequency in your training corpus. If you just used that as your model, it would not be nearly as good. Things like reasoning, a world model, and other heuristics can be used to generate said probabilities internally. All that being said, who is to say that humans aren't just looking at probabilities when they think?

kristopolous 3y ago

That's not how bullshit works. An expertise is needed from the reader when presented with false material. That's why so many people think parody news articles are real. If you have ever gone through any comment thread on the internet about climate change you'll find a large volume of climate deniers citing God, greedy liberals, George Soros, whatever. The people working with these fictions can always just add more fiction to fill in whatever hole. They aren't bound by reality or credulity. Actually understanding global climate systems takes an incredible amount of study. You just can't blurt out a few paragraphs of words and get people to stop believing in Jewish weather machine conspiracies or whatever they're thinking. We would have been done with that by now if this were the case. Dislodging bullshit is hard. Humanity only started figuring out how a few hundred years ago and we're still fucking it up - such as in the replication crisis

bick_nyers 3y ago
Right but I feel like in your argument there is an implicit assumption that the only way to get ChatGPT etc. to understand say a global climate system is by omitting false information entirely. Why can't an LLM extract a signal from the noise? It seems as though it does that already. If you're worried that there is more text on the internet saying that the Earth is flat than there is text saying that the Earth is round, then I understand where you are coming from, but then how would ANYONE come to believe that the Earth is round on their own? By learning how to reason. Edit: My point is that if the LLM can't weed through bullshit properly then how can we expect it to get anything right at all? We can't just feed it a continuously updated list of facts that are true and facts that are false until the end of time.
- eropple 3y ago
  > Right but I feel like in your argument there is an implicit assumption that the only way to get ChatGPT etc. to understand say a global climate system is by omitting false information entirely. Why can't an LLM extract a signal from the noise? It seems as though it does that already. Because it's not thinking, it's looking at probabilities. If you don't prune branches, you've created a disinformation radiator if that information is sufficiently populous within your corpus to be a pretty-good-bet. The best analogy I've come up with is "it's like if you had an objective reality based on Wikipedia edit wars."
  - kristopolous 3y ago
    Exactly. Especially when there's some hustle behind lying to people. Phony medical advice is the classic one here. People do it to sell their sugar pills or do some Peter Popoff style healing touch hustle. Some scams, like MLM, encourage massive volumes of this bullshit to be created and disseminated. The democracy of information does not lead to an increased accuracy based on whatever information spreads the most. The internet is a multi-decade long global proof of this. There's a reason that classrooms have 1 teacher and say 24 students - only 4% of that room is to be trusted with the material. To know if something is true, you need to eventually have some kind of external confirmation. Karl Popper wrote a lot about this. These chat systems can make predictions about the existing corpus but it can't like roll around town and do actual science to confirm things. If all truth must be derived from self-contained systems than you have to make sure your self-contained system isn't bullshit. A game for instance, can have its own rules and consistencies and things can be rationalized through the game. But how much that maps to a reality cannot be concluded strictly from inside the game. For instance, if you want expertise in masonry and construction you need to make sure you aren't just feeding your AI Tetris and SimCity.
  - bick_nyers 3y ago
    Just because it is a probability doesn't mean that it's strictly the probability derived from frequency in your training corpus. If you just used that as your model, it would not be nearly as good. Things like reasoning, a world model, and other heuristics can be used to generate said probabilities internally. All that being said, who is to say that humans aren't just looking at probabilities when they think?