10 ms·

Natural language benchmarks don’t measure AI models’ general knowledge well

65 points by optimalsolver 6y ago

nextos 6y ago
Judea Pearl has been bringing up the lack of causal knowledge in ML very often. He has even posted lots of interesting comments in Andrew Gelman's blog, e.g.: https://statmodeling.stat.columbia.edu/2009/07/05/disputes_about/ https://statmodeling.stat.columbia.edu/2009/07/05/disputes_a... I tend to think that lots of solutions could come from topics like those discussed in this book, with a lot of further development: http://probmods.org/ http://probmods.org/
- Isinlor 6y ago
  Understanding of causality is very likely an emergent property. While extremely important, it's unlikely that we have some hard-coded low level architecture of causal inference in brains. It probably will just arise as a necessity of grounded understanding of the world.
  - mjburgess 6y ago
    I'm not exactly sure what you mean. The conditions for causal inference being possible are pretty clear and have to do with the intentional modification of the local environment. The intention to achieve some new environmental state, and your action to bring it about, is a dynamical activity that enables "deep" model building. Causal inference is not going to be some "module" of the brain... it requires a body. When you place your hand on a hot surface, once, you immediately understand that it is hot. It does not require "induction" (as hume supposed). That is because our body identifies causes. Its therefore pretty trivial to observe no NLP system understands language, or even can understand language, because it lacks this capacity to acquire language semantics via participation in environmental exploration. It has no body. ie., you need to have experienced "on top", "green", etc. to know what "green leaves grow on top of trees" means. There is no meaning in the frequency co-incidence of symbols in text. So no matter how much you are able to reproduce these patterns, they contain no content. The content is in the reader.
    - s_brady 6y ago
      Your comment is interesting but seems to conflate two separate though related areas. The need for a body arises from the embodied cognition school of AI which suggests that intelligence is fundamentally embodied, hence the need for a robot equivalent to a body for truly understanding language. However this does not necessarily have to be related to causality, and counterfactual statements about a causal model. The math behind counterfactuals and causality is actually well understood now (see any of Pearl's books). It does not actually require that a system be embodied, just that the system have some suitable (and correct) causal model of the world. It would of course be amazing to have both in one system, but that is not required. An AI system that understood causality and language could be bootstrapped from causal models supplied by humans - or even other AIs :)
      - mjburgess 6y ago
        I'm conflating them because they are deeply connected. Causal analysis can be performed, via Pearl, on datasets collected for causal analysis. You still need some mechanism to collect the data, ie., the scientist. This requires solving the "relevance" (/framing) problem -- which, in my view, cannot be solved under a congitivist (/computational) theory of mind. "Data" which is relevant to a causal hypothesis isn't selected via inference, the body "selects" it. eg., when my hand is on a hot surface, it's temperature isnt "chosen as the relevant casual variable". The body is the primary solution to the relevance problem. So you can't just "shove in causal math" into a computational system and expect it to grasp anything. I also don't think bootstrapping will take you very far: causal models of, eg., dogs are very deep. ie., we understand their 2d, 3d, skeletal, behavioural, color, sound etc. "dimensions". To say, "the dog was well behaved" requires an extraordinarily deep model of "dog". The only way i see this being built is via play, ie., via hypothetical interaction with an environment -- as we do -- with bodies capable of discerning relevance.
        nextos 6y ago
        >Causal analysis can be performed, via Pearl, on datasets collected for causal analysis. There's a lot of past and current work on causal inference on observational data too.
- jmole 6y ago
  Yes, and text can be written in non-causal ways, which means even if you parse things in order, you can easily end up with a non-causal understanding. “I just saw a huge fish, but then I realized I was out of worms. I should’ve bought more, but I spent $20 on gas instead.” Here the causal order goes from last to first (minus the first clause)
- MarkPNeyer 6y ago
  This sounds like a possible analog to someone who is intelligent but doesn't really understand math or science. This person might be able to BS their way through certain fields, but could never pass a calculus exam.
tsimionescu 6y ago
I'm always amazed that some people think performing statistical analysis (training neural networks) on text can lead to actual intelligence and knowledge about the world, if only we increase the number of model parameters by a few more orders of magnitude (GPT-3 and the likely strategy for GPT-4). Of course, all this without any model of how likely it is that the knowledge is embedded in the text, such as trying to train simple models on descriptions of a simple world. I think it's very likely a model couldn't even learn simple arithmetic (addition, subtraction multiplication, division on the rational numbers, let's say) given all of human writing on arithmetic, nevermind being able to reason about the entire world.
- Isinlor 6y ago
  It does lead to intelligence and knowledge of the world. It leads to somewhat similar understanding of the world as our understanding of 4D, quantum mechanics or environment close to a black hole. Pretty bad one, mostly based on repeating whatever we have read about in popular science literature without any intuitive understanding. GPT-3 understanding of physical world is similarly bad. It's just simply a modality not available to the model. But GPT-3 is able to do few-shots learning. If you show it couple of times how to do something it will try to repeat after you showing some rudimentary understanding of what you are trying to do. It's even able to learn to do simple analogies [0]. GPT-3 is able to do very, very simple arthritics like addition and subtractions with 2-3 numbers. Actually even a lot smaller models trained on arthritics are able to do impressive feats like symbolic integration and solving differential equations. > Deep Learning for Symbolic Mathematics [1]: Neural networks have a reputation for being better at solving statistical or approximate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica. What is true is that GPT-3 is not very good at following instructions. It would not be able to execute an algorithm. So, it would not learn arithmetic from instructions if you would not include explicit examples. [0] https://medium.com/@melaniemitchell.me/can-gpt-3-make-analogies-16436605c446 https://medium.com/@melaniemitchell.me/can-gpt-3-make-analog... [1] https://arxiv.org/abs/1912.01412 https://arxiv.org/abs/1912.01412
skybrian 6y ago
It seems like there are not many common-knowledge questions that you can't find answers to on the Internet? Most of us rely on web search these days. Also, making up answers isn't necessarily a good skill to train for if accuracy is needed. It would be more useful to quote and cite the source.
Veedrac 6y ago
The article doesn't give examples, but the paper does. Here are a few examples of question overlap, for context. Test Question: who plays max voice in a goofy movie Train Question: who does max voice in a goofy movie Answer: Jason Marsden Test Question: when will the 2018 oscar nominations be announced Train Question: when are the oscar nominations for 2018 announced Answer: January 23 2018 Test Question: who has scored more goals in the premier league Train Question: most goals scored by a premier league player Answer: Alan Shearer Test Question: where are the cones in the eye located Train Question: where are cone cells located in the eye Answer: retina Test Question: who led the conquest of the incas in south america Train Question: conquistador who defeated the incan empire in peru Answer: francisco pizarro It makes sense that questions like these are especially susceptible to brute memorization. These are certainly bugs in the benchmark—and, in fact, about a third of questions in each dataset have this issue. The paper also considers answer overlap; that is, when the answer to the question also occurs in the training set. This alone does not imply memorization, but it does open the door for some shortcuts. Some examples from the paper are: Open Natural Questions Duplicated: Phil Simms, Brian Johnson, 8, the Indians, the 1830s Unique: Cloves, Matt Monro, 1,020 – 1,080 kg, Hermann Ebbinghaus, Matt Flinders TriviaQA Duplicated: David Bowie, Battle of camlann, Heligoland, Henry VII, Niagra Falls Unique: Death in the afternoon, Clash of the Titans, ice-cream sundae, Camshaft, Cumberland WebQuestions Duplicated: Harvard, Alderaan, India, 2011, Zeus Unique: Queen Victoria, Braslia, Paddington, Tom Corbett, Gary It's less obvious how to treat answer overlap. In particular, removing answer overlap might bias the dataset towards harder questions. Some of the reduction in model scores is presumably because models have to work harder to understand the question, as merely looking for topical similarities will be ineffective, but it also removes questions whose answers are general enough to apply to many questions, like ‘8’, ‘the 1930s’, ‘Harvard’, ‘2011’, etc. This means that reductions in the score don't clearly say how much cheating happened. However, the success of the BERT-based Nearest Neighbor model, which retrieves the answer of the most semantically similar fine-tuning sample, scores similarly to BART, which seems much too high for comfort. It should be noted that a lot of the discussions about causal knowledge and intelligence aren't too relevant to these questions, as they are largely tests of memory or retrieval. It is expected that these models answer the questions by searching their training data or their document index. The issue is that their training data isn't meant to contain copies of the test questions, just the information sufficient to answer them.
ninjin 6y ago
Second author here with a link to the arXiv paper: https://arxiv.org/abs/2008.02637 https://arxiv.org/abs/2008.02637 Although I have to say, VentureBeat did much better than most media outlets I have seen writing about current research and what they write is not only accurate but also largely devoid of hype. Perhaps we actually managed to “keep the hype down” as we intended when writing this piece? I will check in on this post now and then if you have questions and see if the first author is interested in joining when he wakes up as he really did all the legwork for this one.
- nl 6y ago
  Nice paper and important work. Given that nearest-neighbor outperforms on closed book, is it reasonable to suspect the model is doing NN itself internally (which would explain the good performance on close duplicates?) And if this is the case do you think training-time processing of data to attempt to move convert it to question/answer form data rather than raw QA would be a reasonable approach towards tackling this?
  - patrick-lewis 6y ago
    Hi, first author here. > Given that nearest-neighbor outperforms on closed book, is it reasonable to suspect the model is doing NN itself internally (which would explain the good performance on close duplicates?) I think this is definitely the case for the BART model. It is essentially acting as a QA-pair memorizer over the training data, and at test time, it just matches the question onto those seen at training time. Note that the T5-11B+SSM closed-book model was able to do a little better on NQ, so very large models with task-specific pretraining objectives do seem to do something slightly more interesting than just NN, but still really struggle in some settings. > And if this is the case do you think training-time processing of data to attempt to move convert it to question/answer form data rather than raw QA would be a reasonable approach towards tackling this? Great question! Converting sentences into a series of QA pairs is something we're really interested in. The T5-11B+SSM model we evaluate in the paper uses a special "Salient span masking" pretraining objective that does this to some extent (only mask words at pretraining time that are likely to be "answers" to factual questions), so in essence the pretraining task becomes pretty standard cloze-question answering, and they find that leads to better downstream results (https://arxiv.org/abs/2002.08910 https://arxiv.org/abs/2002.08910)

65 points by optimalsolver 6y ago

Judea Pearl has been bringing up the lack of causal knowledge in ML very often. He has even posted lots of interesting comments in Andrew Gelman's blog, e.g.: https://statmodeling.stat.columbia.edu/2009/07/05/disputes_about/ https://statmodeling.stat.columbia.edu/2009/07/05/disputes_a... I tend to think that lots of solutions could come from topics like those discussed in this book, with a lot of further development: http://probmods.org/ http://probmods.org/

Isinlor 6y ago
Understanding of causality is very likely an emergent property. While extremely important, it's unlikely that we have some hard-coded low level architecture of causal inference in brains. It probably will just arise as a necessity of grounded understanding of the world.
- mjburgess 6y ago
  I'm not exactly sure what you mean. The conditions for causal inference being possible are pretty clear and have to do with the intentional modification of the local environment. The intention to achieve some new environmental state, and your action to bring it about, is a dynamical activity that enables "deep" model building. Causal inference is not going to be some "module" of the brain... it requires a body. When you place your hand on a hot surface, once, you immediately understand that it is hot. It does not require "induction" (as hume supposed). That is because our body identifies causes. Its therefore pretty trivial to observe no NLP system understands language, or even can understand language, because it lacks this capacity to acquire language semantics via participation in environmental exploration. It has no body. ie., you need to have experienced "on top", "green", etc. to know what "green leaves grow on top of trees" means. There is no meaning in the frequency co-incidence of symbols in text. So no matter how much you are able to reproduce these patterns, they contain no content. The content is in the reader.
  - s_brady 6y ago
    Your comment is interesting but seems to conflate two separate though related areas. The need for a body arises from the embodied cognition school of AI which suggests that intelligence is fundamentally embodied, hence the need for a robot equivalent to a body for truly understanding language. However this does not necessarily have to be related to causality, and counterfactual statements about a causal model. The math behind counterfactuals and causality is actually well understood now (see any of Pearl's books). It does not actually require that a system be embodied, just that the system have some suitable (and correct) causal model of the world. It would of course be amazing to have both in one system, but that is not required. An AI system that understood causality and language could be bootstrapped from causal models supplied by humans - or even other AIs :)
    - mjburgess 6y ago
      I'm conflating them because they are deeply connected. Causal analysis can be performed, via Pearl, on datasets collected for causal analysis. You still need some mechanism to collect the data, ie., the scientist. This requires solving the "relevance" (/framing) problem -- which, in my view, cannot be solved under a congitivist (/computational) theory of mind. "Data" which is relevant to a causal hypothesis isn't selected via inference, the body "selects" it. eg., when my hand is on a hot surface, it's temperature isnt "chosen as the relevant casual variable". The body is the primary solution to the relevance problem. So you can't just "shove in causal math" into a computational system and expect it to grasp anything. I also don't think bootstrapping will take you very far: causal models of, eg., dogs are very deep. ie., we understand their 2d, 3d, skeletal, behavioural, color, sound etc. "dimensions". To say, "the dog was well behaved" requires an extraordinarily deep model of "dog". The only way i see this being built is via play, ie., via hypothetical interaction with an environment -- as we do -- with bodies capable of discerning relevance.
      - nextos 6y ago
        >Causal analysis can be performed, via Pearl, on datasets collected for causal analysis. There's a lot of past and current work on causal inference on observational data too.
jmole 6y ago
Yes, and text can be written in non-causal ways, which means even if you parse things in order, you can easily end up with a non-causal understanding. “I just saw a huge fish, but then I realized I was out of worms. I should’ve bought more, but I spent $20 on gas instead.” Here the causal order goes from last to first (minus the first clause)
MarkPNeyer 6y ago
This sounds like a possible analog to someone who is intelligent but doesn't really understand math or science. This person might be able to BS their way through certain fields, but could never pass a calculus exam.

tsimionescu 6y ago

I'm always amazed that some people think performing statistical analysis (training neural networks) on text can lead to actual intelligence and knowledge about the world, if only we increase the number of model parameters by a few more orders of magnitude (GPT-3 and the likely strategy for GPT-4). Of course, all this without any model of how likely it is that the knowledge is embedded in the text, such as trying to train simple models on descriptions of a simple world. I think it's very likely a model couldn't even learn simple arithmetic (addition, subtraction multiplication, division on the rational numbers, let's say) given all of human writing on arithmetic, nevermind being able to reason about the entire world.

Isinlor 6y ago
It does lead to intelligence and knowledge of the world. It leads to somewhat similar understanding of the world as our understanding of 4D, quantum mechanics or environment close to a black hole. Pretty bad one, mostly based on repeating whatever we have read about in popular science literature without any intuitive understanding. GPT-3 understanding of physical world is similarly bad. It's just simply a modality not available to the model. But GPT-3 is able to do few-shots learning. If you show it couple of times how to do something it will try to repeat after you showing some rudimentary understanding of what you are trying to do. It's even able to learn to do simple analogies [0]. GPT-3 is able to do very, very simple arthritics like addition and subtractions with 2-3 numbers. Actually even a lot smaller models trained on arthritics are able to do impressive feats like symbolic integration and solving differential equations. > Deep Learning for Symbolic Mathematics [1]: Neural networks have a reputation for being better at solving statistical or approximate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica. What is true is that GPT-3 is not very good at following instructions. It would not be able to execute an algorithm. So, it would not learn arithmetic from instructions if you would not include explicit examples. [0] https://medium.com/@melaniemitchell.me/can-gpt-3-make-analogies-16436605c446 https://medium.com/@melaniemitchell.me/can-gpt-3-make-analog... [1] https://arxiv.org/abs/1912.01412 https://arxiv.org/abs/1912.01412

skybrian 6y ago

It seems like there are not many common-knowledge questions that you can't find answers to on the Internet? Most of us rely on web search these days. Also, making up answers isn't necessarily a good skill to train for if accuracy is needed. It would be more useful to quote and cite the source.

Veedrac 6y ago

The article doesn't give examples, but the paper does. Here are a few examples of question overlap, for context. Test Question: who plays max voice in a goofy movie Train Question: who does max voice in a goofy movie Answer: Jason Marsden Test Question: when will the 2018 oscar nominations be announced Train Question: when are the oscar nominations for 2018 announced Answer: January 23 2018 Test Question: who has scored more goals in the premier league Train Question: most goals scored by a premier league player Answer: Alan Shearer Test Question: where are the cones in the eye located Train Question: where are cone cells located in the eye Answer: retina Test Question: who led the conquest of the incas in south america Train Question: conquistador who defeated the incan empire in peru Answer: francisco pizarro It makes sense that questions like these are especially susceptible to brute memorization. These are certainly bugs in the benchmark—and, in fact, about a third of questions in each dataset have this issue. The paper also considers answer overlap; that is, when the answer to the question also occurs in the training set. This alone does not imply memorization, but it does open the door for some shortcuts. Some examples from the paper are: Open Natural Questions Duplicated: Phil Simms, Brian Johnson, 8, the Indians, the 1830s Unique: Cloves, Matt Monro, 1,020 – 1,080 kg, Hermann Ebbinghaus, Matt Flinders TriviaQA Duplicated: David Bowie, Battle of camlann, Heligoland, Henry VII, Niagra Falls Unique: Death in the afternoon, Clash of the Titans, ice-cream sundae, Camshaft, Cumberland WebQuestions Duplicated: Harvard, Alderaan, India, 2011, Zeus Unique: Queen Victoria, Braslia, Paddington, Tom Corbett, Gary It's less obvious how to treat answer overlap. In particular, removing answer overlap might bias the dataset towards harder questions. Some of the reduction in model scores is presumably because models have to work harder to understand the question, as merely looking for topical similarities will be ineffective, but it also removes questions whose answers are general enough to apply to many questions, like ‘8’, ‘the 1930s’, ‘Harvard’, ‘2011’, etc. This means that reductions in the score don't clearly say how much cheating happened. However, the success of the BERT-based Nearest Neighbor model, which retrieves the answer of the most semantically similar fine-tuning sample, scores similarly to BART, which seems much too high for comfort. It should be noted that a lot of the discussions about causal knowledge and intelligence aren't too relevant to these questions, as they are largely tests of memory or retrieval. It is expected that these models answer the questions by searching their training data or their document index. The issue is that their training data isn't meant to contain copies of the test questions, just the information sufficient to answer them.

ninjin 6y ago

Second author here with a link to the arXiv paper: https://arxiv.org/abs/2008.02637 https://arxiv.org/abs/2008.02637 Although I have to say, VentureBeat did much better than most media outlets I have seen writing about current research and what they write is not only accurate but also largely devoid of hype. Perhaps we actually managed to “keep the hype down” as we intended when writing this piece? I will check in on this post now and then if you have questions and see if the first author is interested in joining when he wakes up as he really did all the legwork for this one.

nl 6y ago
Nice paper and important work. Given that nearest-neighbor outperforms on closed book, is it reasonable to suspect the model is doing NN itself internally (which would explain the good performance on close duplicates?) And if this is the case do you think training-time processing of data to attempt to move convert it to question/answer form data rather than raw QA would be a reasonable approach towards tackling this?
- patrick-lewis 6y ago
  Hi, first author here. > Given that nearest-neighbor outperforms on closed book, is it reasonable to suspect the model is doing NN itself internally (which would explain the good performance on close duplicates?) I think this is definitely the case for the BART model. It is essentially acting as a QA-pair memorizer over the training data, and at test time, it just matches the question onto those seen at training time. Note that the T5-11B+SSM closed-book model was able to do a little better on NQ, so very large models with task-specific pretraining objectives do seem to do something slightly more interesting than just NN, but still really struggle in some settings. > And if this is the case do you think training-time processing of data to attempt to move convert it to question/answer form data rather than raw QA would be a reasonable approach towards tackling this? Great question! Converting sentences into a series of QA pairs is something we're really interested in. The T5-11B+SSM model we evaluate in the paper uses a special "Salient span masking" pretraining objective that does this to some extent (only mask words at pretraining time that are likely to be "answers" to factual questions), so in essence the pretraining task becomes pretty standard cloze-question answering, and they find that leads to better downstream results (https://arxiv.org/abs/2002.08910 https://arxiv.org/abs/2002.08910)