10 ms·

Pretraining data enables narrow selection capabilities in transformer models

65 points by hislaziness 3y ago

ldjkfkdsjnv 3y ago
My uneducated opinion is that this paper is bullocks. Maybe they are looking at deeper mathemtical results, instead of every day tasks. But every single day I am using OpenAI GPT4 to handle novel tasks. I am working on a traditional saas vertical, except with a pure chatbot. The model works, is able to understand which function to call, to extract which parameters, and to know when the inputs will not work. Sure, if you ask it to do some extraneous task, it fails. Google/Deep Mind need to start showing up with some working results. Where. are. the. models. google.
- valine 3y ago
  I use it frequently with my own custom UI framework. It’s never seen my framework before but it can output new, useable code with just a few examples. If that’s not generalization I don’t know what is.
  - vidarh 3y ago
    I've given it descriptions of non-existent "franken-languages" composed by telling it to imagine taking programming language A and adding various features I want to explore to it, and then had it correctly symbolically reason about a program written in this hypothetical language that doesn't exist anywhere, so yeah, the notion it doesn't generalize to at least some degree is nonsense, but note this involved tests on a GPT2 scale model so it's not very surprising they had poor results. That said, even GPT4 certainly has pretty significant limitations on what it manages to reason about. But without comparing their capabilities in other aspects, arguably so do most humans. We tend to force our way past those limitations by learning incrementally by doing over and over. Current models don't get that luxury without complicated fine-tuning steps, so if anything what should surprise us is how well they do with the limitation of only context to act as short-term memory.
    - simbolit 3y ago
      The way I understand the paper, his is still a domain-specific tasks, and it is interpolating. What (according to the paper) transformer models are bad at, is out-of-domain extrapolation (which given that modern models are basically trained on all the internet is kinda uncommon).
      - vidarh 3y ago
        If so, then I'd argue that most of humans' "out of domain extrapolation" is domain-specific interpolation as well, and that most humans would struggle to meet the bar.
        simbolit 3y ago
        No disagreement. This is only relevant for the AI-bro fantasy of AI becoming exponentially smarter than humans.
        vidarh 3y ago
        I don't think it's clear whether or not extrapolation in a sense that'd clear that bar is of much relevance to becoming smart at all.
        simbolit 3y ago
        The AI-bro fantasy is of AI building a smarter AI, which then builds an even smarter one. If this isn't extrapolation, does it exist?
        vidarh 3y ago
        I'm not sure whether that requires extrapolation or not. It depends on what you consider extrapolation. I'd be pretty sceptical of the notion that it requires extrapolation on the basis of a use of the term as extreme as the one above. (And btw., if that is what you consider the "AI-bro fantasy" then I'm firmly in that camp. There is no logical reason to assume that's not possible; short of identifying a non-deterministic, non-materialistic source of intelligence in human brains that violates known physics and that we can't emulate, it's just a question of when, not if, and the reason for not going full tilt on that right now is down to a variant of the Wait Calculation, not a need for intelligence to kick it off - that is, trying to kickstart that now is likely to be more expensive and not be any faster than trying to kickstart it next year, and so on, and the question is guessing at when that stops being true)
  - rdedev 3y ago
    The issue is though the the line between in domain and out of domain is fuzzy. This sort of means that generalization is in a continum. Chatgpt has seen enough UI framework code that it can interpolate concepts. This is a form of generalization but people would be looking for a lot more. I guess a better way to check generalization capability is to train the model on just C++ and then see how much it can do stuff in python using only few shot examples. Another important thing to keep in mind is one paper(wish I could remember which one it was) that showed even larger scale llms have trouble understanding that A=B is same as B=A if they have not seen A or B before
    - valine 3y ago
      Going from C++ to python is a little unfair. How many examples would a human need to learn python coming from C++? Its probably more than you can fit in a LLM context window.
      - telotortium 3y ago
        Not with GPT-4-Turbo 128k, at least if attention works as well at that context size as on smaller ones.
      - rdedev 3y ago
        I know it's unfair compared to a human but I'm more interested in how much it can do. Like what level of leetcode problems can it solve and how well does it use concepts presented in the few shot applications. The whole point is to establish an upper limit on it's generalization power instead of comparing it to a human
    - sandkoan 3y ago
      For anyone else interested, the paper he's referring to is "The Reversal Curse": https://arxiv.org/abs/2309.12288 https://arxiv.org/abs/2309.12288.
  - idontpost 3y ago
    [dead]
  - SoftTalker 3y ago
    "If you've seen one custom UI framework....."
  - smohare 3y ago
    [dead]
- haltist 3y ago
  Which part do you think is incorrect?
- hfjjbf 3y ago
  [dead]
- dontwearitout 3y ago
  An aside, but do you find the quality improvements in GPT-4 are necessary for your daily use, or is 3.5 80% as good?
  - sharemywin 3y ago
    I upgraded because I wanted to see what it could do with a screen shot of a web page. I had it describe the page and create an html version of the page. it wasn't horrible.
wavelander 3y ago
I'm not sure folks who're putting out strong takes based on this have read this paper. This paper uses GPT-2 transformer scale, on sinusoidal data: >We trained a decoder-only Transformer [7] model of GPT-2 scale implemented in the Jax based machine learning framework, Pax4 with 12 layers, 8 attention heads, and a 256-dimensional embedding space (9.5M parameters) as our base configuration [4]. > Building on previous work, we investigate this question in a controlled setting, where we study transformer models trained on sequences of (x,f(x)) pairs rather than natural language. Nowhere near definitive or conclusive. Not sure why this is news outside of the Twitter-techno-pseudo-academic-influencer bubble.
- andy99 3y ago
  It would be news is somebody showed transformers could generalize beyond the training data. Deep learning models generally cannot, so it's not a surprise this holds for transformers.
  - GaggiX 3y ago
    It depends on what does "generalize beyond the training data" means. If I invent a new programming language and I teach (in-context) the language to the model and it's able to use it to solve many tasks, is it generalizing beyond the training data?
    - andy99 3y ago
      No. The way I'd look at it is that generalization or specifically extrapolation would mean that different features are needed to make a prediction (here, the next token) than what is seen in the training data. Something like a made up language could still result in the same patterns being relevant. That's why out-of-distribution research often uses mathematical extrapolation as a task.
      - famouswaffles 3y ago
        "Supercharged Interpolation" is not something that actually exists. Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485 What you're asking for is not "generalization" but magic and humans would also fail.
simbolit 3y ago
TLDR: transformer models (on gpt2 scale) are great (near-optimal) at interpolating between the cases given in (pre-)training, but as soon as we leave the training domain fail at extrapolation. Impressive results may be more due to the wide breadth of (pre-)training data, and less due to generalization ability.
- deleted 3y ago
  [deleted]
simbolit 3y ago
Why are transformer models so bad at math? They often fail at simple addition.
- jjtheblunt 3y ago
  Don't they essentially memorize all examples they've seen, which contrasts with understanding how to solve problems as in addition with carries on digits etc?
  - tsunamifury 3y ago
    No, stop repeating this. They use examples to learn and attempt to generalize with on-the-fly modeling.
    - jjtheblunt 3y ago
      Are you claiming the on the fly modeling can learn or reproduce multistep algorithms, like simple addition that the parent comment asked about, and which i asked about above?
      - vidarh 3y ago
        I just fed it this: --- Here is a grammar for a made up language: sentence = "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). verb = "ab" | "pom" object = "foo" | "bar" Construct a few sentences, and show the sentence so far, and which rule and step of the rule you are currently applying. E.g. sentence= ""; rule=sentence; step="sob". --- Notice I did not explain the BNF syntax, which is reasonable as it'll have plenty of that in its training data. Notice that I also "forgot" to give it a production for the "subject" rule - leaving it with two choices: It can give up and tell me off, or it can do as best it can and let me know it's had to improvise. Here's the first sentence construction: --- First Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: (choosing object as a subject for simplicity) "foo". sentence = "sob foo"; Rule: verb; Step: "ab". sentence = "sob foo ab"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ob". sentence = "sob foo ab ob"; Rule: object; Step: "bar". sentence = "sob foo ab ob bar"; (Complete Sentence) --- It has correctly applied the rules I did give, and opted to "fill in" "object" as a replacement for my missing "subject" rule. There are few enough rules this could be a fluke, but it's a better choice to guess at that substitution than verb or sentence, though hardly takes a genius. Second: --- Second Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: "bar". sentence = "sob bar"; Rule: verb; Step: "pom". sentence = "sob bar pom"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ad". sentence = "sob bar pom ad"; Rule: sentence; Step: "sob" (this starts a nested sentence). sentence = "sob bar pom ad sob"; Rule: subject; Step: "foo". sentence = "sob bar pom ad sob foo"; Rule: verb; Step: "ab". sentence = "sob bar pom ad sob foo ab"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ob". sentence = "sob bar pom ad sob foo ab ob"; Rule: object; Step: "foo". sentence = "sob bar pom ad sob foo ab ob foo"; (Complete Sentence) --- This is also correct, and it correctly points out that the arm it picked triggered a nested application of the `sentence` rule. It stumbled on the third attempt: --- Third Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: "foo". sentence = "sob foo"; Rule: verb; Step: "pom". sentence = "sob foo pom"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ploj". sentence = "sob foo pom ploj"; (Complete Sentence) --- Note that there is a possible interpretation of the "|" operator which would make its sentence valid, so "technically" it could be, but it would involve (ver "ob" object | verb "ad" sentence | "ploj") to be interpreted as (verb (("ob object)|("ad" sentence)|"ploj")) and that's contrived. So to answer you: It can, and it does, but not always consistently. One can choose to be disappointed it's not perfect, but for just handing it a grammar and asking it to reason about it with no example text or chances to "practice" whatsoever, I think it's doing quite well. I've interviewed many software developers who'd struggle with a task like this. When prodded about the third example, it picks a more reasonable interpretation of the "|", though it's explanation starts out with an error, but then given a correct explanation: --- Q: How do you figure this? The full rule is "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). How do you figure the verb is allowed without being followed by either "ad" or "ob"? A: Upon closer examination, you are correct to point out that the structure following "sob" subject must indeed involve a verb according to the rule provided. The full rule: "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). indicates that the sentence must start with "sob", followed by a subject, and then either: a verb followed by "ob" and an object, a verb followed by "ad" and another sentence, or the standalone term "ploj". --- It goes on to give two correct alternatives to the erroneous one: --- * "sob foo ploj" — where "ploj" follows directly after the subject without a verb, or * "sob foo ab ob bar" — where "ab" is the verb followed by "ob" and "bar" as the object. --- While it's not perfect, it can very clearly "learn and reproduce multistep algorithms" to some extent. It also clearly needs to get better at it.
MeImCounting 3y ago
The one thing is that they seem to be using relatively small models. This may be a really damning result but I was under the impression that any generalization capabilities of LLMs appear in a non-linear fashion when you increase the parameter count to the tens of billions/trillions as in GPT4. It would be interesting if they could recreate the same experiment with a much larger model. Unfortunately I dont think thats likely to happen because of the resources required to train such models and the anti-open-source hysteria likely preventing larger models from being made publicly available much less the data they were trained on. Imagine that, stifling research and fearmongering reduces the usefulness of the science that does manage to get done.
mediaman 3y ago
There have been two criticisms of this paper floating around. 1. The test mechanism is to use prediction of sinusoidal series. While it's certainly possible to train transformers on mathematical functions, it's not clear why findings from a model trained on sinusoidal functions would generalize into the domain of written human language (which is ironic, given the paper's topic). 2. Even if it were true that these models don't generalize beyond their training, large LLMs' training corpus is basically all of written human knowledge. So then the goalpost has been moved to "well, they won't push the frontier of human knowledge forward," which seems to be a much diminished claim, since the vast majority of humans are also not pushing the frontier of human knowledge forward and instead use existing human knowledge to accomplish their daily goals.
- hansvm 3y ago
  You're using the idea of "all of human knowledge" differently in the two places it appears, and the gap in those definitions weakens the claim a bit. LLMs are trained on a tiny subset of the written human knowledge which we've proven to probably not be garbage and which is nicely formatted in simple text formats and which was published without too many paywalls on the web and on and on and on. It's a lot, and it definitely includes enough facts that no one person knows all the things the LLM "knows", but the average child knows plenty of things which never made it into that sort of a corpus. Yes, it's probably true that the vast majority of humans are also not pushing the frontier of human knowledge forward, but the vast majority of humans are working with a slightly different (partially overlapping) set of information than what the LLMs see.
  - Isomorpheus 3y ago
    Can you name three things an average child knows which GPT-4 doesn't?
    - abeppu 3y ago
      How to drink from a cup without dribbling. The feeling of sunshine on one's skin. The experience of the passage of time.
pizza 3y ago
FWIW the paper title is focuses on quite a different conclusion than the submission title: Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models
lsh123 3y ago
Current AI models are approximation functions with huge number of parameters. These approximation functions are reasonably good at interpolation, meh at extrapolation, and have nothing to do with generalization.
- famouswaffles 3y ago
  Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485 "Supercharged Interpolation" is not a real thing. They generalize fine when the data incentivizes that.
  - lsh123 3y ago
    You always can extrapolate. E.g. linear approximation for x^2 by 2 points will extrapolate reasonably well around these 2 points but will be bad with x -> +/- infinity. Similarly, there are examples where GPT invented legal cases when asked to create a legal brief.
    - samus 3y ago
      I asked GPT 3.5 how to access the Unihan character database[0] using the ICU4J library[1]. GPT 3.5 suggested to use the `com.ibm.icu.util.Unihan` class. It actually doesn't exist, but after pointing that out to it, GPT showed me how to parse the raw database files. [0]: https://www.unicode.org/charts/unihan.html https://www.unicode.org/charts/unihan.html [1]: https://unicode-org.github.io/icu/userguide/icu4j/ https://unicode-org.github.io/icu/userguide/icu4j/
Der_Einzige 3y ago
We humans don't even know when we are doing real extrapolation, and the vast majority of humans are interpolating. I bet many do nothing but interpolate their whole lives. So - and I say this as someone who writes NLP papers too - who cares?
datadrivenangel 3y ago
Overfitting much?
famouswaffles 3y ago
"Generalization" is always a data problem. If you trained it on one function class, of course that's all it learned to do. That's all it ever saw! If you want to learn arbitrary function classes to some degree, the solution is simple. Train it on many different function classes. Untrained models are as blank slate as you could possibly imagine. They're not even comparable to new born humans with millions of years of evolution baked in. The data you feed them is their world. Their only world.

65 points by hislaziness 3y ago

My uneducated opinion is that this paper is bullocks. Maybe they are looking at deeper mathemtical results, instead of every day tasks. But every single day I am using OpenAI GPT4 to handle novel tasks. I am working on a traditional saas vertical, except with a pure chatbot. The model works, is able to understand which function to call, to extract which parameters, and to know when the inputs will not work. Sure, if you ask it to do some extraneous task, it fails. Google/Deep Mind need to start showing up with some working results. Where. are. the. models. google.

valine 3y ago
I use it frequently with my own custom UI framework. It’s never seen my framework before but it can output new, useable code with just a few examples. If that’s not generalization I don’t know what is.
- vidarh 3y ago
  I've given it descriptions of non-existent "franken-languages" composed by telling it to imagine taking programming language A and adding various features I want to explore to it, and then had it correctly symbolically reason about a program written in this hypothetical language that doesn't exist anywhere, so yeah, the notion it doesn't generalize to at least some degree is nonsense, but note this involved tests on a GPT2 scale model so it's not very surprising they had poor results. That said, even GPT4 certainly has pretty significant limitations on what it manages to reason about. But without comparing their capabilities in other aspects, arguably so do most humans. We tend to force our way past those limitations by learning incrementally by doing over and over. Current models don't get that luxury without complicated fine-tuning steps, so if anything what should surprise us is how well they do with the limitation of only context to act as short-term memory.
  - simbolit 3y ago
    The way I understand the paper, his is still a domain-specific tasks, and it is interpolating. What (according to the paper) transformer models are bad at, is out-of-domain extrapolation (which given that modern models are basically trained on all the internet is kinda uncommon).
    - vidarh 3y ago
      If so, then I'd argue that most of humans' "out of domain extrapolation" is domain-specific interpolation as well, and that most humans would struggle to meet the bar.
      - simbolit 3y ago
        No disagreement. This is only relevant for the AI-bro fantasy of AI becoming exponentially smarter than humans.
        vidarh 3y ago
        I don't think it's clear whether or not extrapolation in a sense that'd clear that bar is of much relevance to becoming smart at all.
        simbolit 3y ago
        The AI-bro fantasy is of AI building a smarter AI, which then builds an even smarter one. If this isn't extrapolation, does it exist?
        vidarh 3y ago
        I'm not sure whether that requires extrapolation or not. It depends on what you consider extrapolation. I'd be pretty sceptical of the notion that it requires extrapolation on the basis of a use of the term as extreme as the one above. (And btw., if that is what you consider the "AI-bro fantasy" then I'm firmly in that camp. There is no logical reason to assume that's not possible; short of identifying a non-deterministic, non-materialistic source of intelligence in human brains that violates known physics and that we can't emulate, it's just a question of when, not if, and the reason for not going full tilt on that right now is down to a variant of the Wait Calculation, not a need for intelligence to kick it off - that is, trying to kickstart that now is likely to be more expensive and not be any faster than trying to kickstart it next year, and so on, and the question is guessing at when that stops being true)
- rdedev 3y ago
  The issue is though the the line between in domain and out of domain is fuzzy. This sort of means that generalization is in a continum. Chatgpt has seen enough UI framework code that it can interpolate concepts. This is a form of generalization but people would be looking for a lot more. I guess a better way to check generalization capability is to train the model on just C++ and then see how much it can do stuff in python using only few shot examples. Another important thing to keep in mind is one paper(wish I could remember which one it was) that showed even larger scale llms have trouble understanding that A=B is same as B=A if they have not seen A or B before
  - valine 3y ago
    Going from C++ to python is a little unfair. How many examples would a human need to learn python coming from C++? Its probably more than you can fit in a LLM context window.
    - telotortium 3y ago
      Not with GPT-4-Turbo 128k, at least if attention works as well at that context size as on smaller ones.
    - rdedev 3y ago
      I know it's unfair compared to a human but I'm more interested in how much it can do. Like what level of leetcode problems can it solve and how well does it use concepts presented in the few shot applications. The whole point is to establish an upper limit on it's generalization power instead of comparing it to a human
  - sandkoan 3y ago
    For anyone else interested, the paper he's referring to is "The Reversal Curse": https://arxiv.org/abs/2309.12288 https://arxiv.org/abs/2309.12288.
- idontpost 3y ago
  [dead]
- SoftTalker 3y ago
  "If you've seen one custom UI framework....."
- smohare 3y ago
  [dead]
haltist 3y ago
Which part do you think is incorrect?
hfjjbf 3y ago
[dead]
dontwearitout 3y ago
An aside, but do you find the quality improvements in GPT-4 are necessary for your daily use, or is 3.5 80% as good?
- sharemywin 3y ago
  I upgraded because I wanted to see what it could do with a screen shot of a web page. I had it describe the page and create an html version of the page. it wasn't horrible.

wavelander 3y ago

I'm not sure folks who're putting out strong takes based on this have read this paper. This paper uses GPT-2 transformer scale, on sinusoidal data: >We trained a decoder-only Transformer [7] model of GPT-2 scale implemented in the Jax based machine learning framework, Pax4 with 12 layers, 8 attention heads, and a 256-dimensional embedding space (9.5M parameters) as our base configuration [4]. > Building on previous work, we investigate this question in a controlled setting, where we study transformer models trained on sequences of (x,f(x)) pairs rather than natural language. Nowhere near definitive or conclusive. Not sure why this is news outside of the Twitter-techno-pseudo-academic-influencer bubble.

andy99 3y ago
It would be news is somebody showed transformers could generalize beyond the training data. Deep learning models generally cannot, so it's not a surprise this holds for transformers.
- GaggiX 3y ago
  It depends on what does "generalize beyond the training data" means. If I invent a new programming language and I teach (in-context) the language to the model and it's able to use it to solve many tasks, is it generalizing beyond the training data?
  - andy99 3y ago
    No. The way I'd look at it is that generalization or specifically extrapolation would mean that different features are needed to make a prediction (here, the next token) than what is seen in the training data. Something like a made up language could still result in the same patterns being relevant. That's why out-of-distribution research often uses mathematical extrapolation as a task.
    - famouswaffles 3y ago
      "Supercharged Interpolation" is not something that actually exists. Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485 What you're asking for is not "generalization" but magic and humans would also fail.

simbolit 3y ago

TLDR: transformer models (on gpt2 scale) are great (near-optimal) at interpolating between the cases given in (pre-)training, but as soon as we leave the training domain fail at extrapolation. Impressive results may be more due to the wide breadth of (pre-)training data, and less due to generalization ability.

deleted 3y ago
[deleted]

simbolit 3y ago

Why are transformer models so bad at math? They often fail at simple addition.

jjtheblunt 3y ago
Don't they essentially memorize all examples they've seen, which contrasts with understanding how to solve problems as in addition with carries on digits etc?
- tsunamifury 3y ago
  No, stop repeating this. They use examples to learn and attempt to generalize with on-the-fly modeling.
  - jjtheblunt 3y ago
    Are you claiming the on the fly modeling can learn or reproduce multistep algorithms, like simple addition that the parent comment asked about, and which i asked about above?
    - vidarh 3y ago
      I just fed it this: --- Here is a grammar for a made up language: sentence = "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). verb = "ab" | "pom" object = "foo" | "bar" Construct a few sentences, and show the sentence so far, and which rule and step of the rule you are currently applying. E.g. sentence= ""; rule=sentence; step="sob". --- Notice I did not explain the BNF syntax, which is reasonable as it'll have plenty of that in its training data. Notice that I also "forgot" to give it a production for the "subject" rule - leaving it with two choices: It can give up and tell me off, or it can do as best it can and let me know it's had to improvise. Here's the first sentence construction: --- First Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: (choosing object as a subject for simplicity) "foo". sentence = "sob foo"; Rule: verb; Step: "ab". sentence = "sob foo ab"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ob". sentence = "sob foo ab ob"; Rule: object; Step: "bar". sentence = "sob foo ab ob bar"; (Complete Sentence) --- It has correctly applied the rules I did give, and opted to "fill in" "object" as a replacement for my missing "subject" rule. There are few enough rules this could be a fluke, but it's a better choice to guess at that substitution than verb or sentence, though hardly takes a genius. Second: --- Second Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: "bar". sentence = "sob bar"; Rule: verb; Step: "pom". sentence = "sob bar pom"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ad". sentence = "sob bar pom ad"; Rule: sentence; Step: "sob" (this starts a nested sentence). sentence = "sob bar pom ad sob"; Rule: subject; Step: "foo". sentence = "sob bar pom ad sob foo"; Rule: verb; Step: "ab". sentence = "sob bar pom ad sob foo ab"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ob". sentence = "sob bar pom ad sob foo ab ob"; Rule: object; Step: "foo". sentence = "sob bar pom ad sob foo ab ob foo"; (Complete Sentence) --- This is also correct, and it correctly points out that the arm it picked triggered a nested application of the `sentence` rule. It stumbled on the third attempt: --- Third Sentence Construction: sentence = ""; Rule: sentence; Step: "sob". sentence = "sob"; Rule: subject; Step: "foo". sentence = "sob foo"; Rule: verb; Step: "pom". sentence = "sob foo pom"; Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ploj". sentence = "sob foo pom ploj"; (Complete Sentence) --- Note that there is a possible interpretation of the "|" operator which would make its sentence valid, so "technically" it could be, but it would involve (ver "ob" object | verb "ad" sentence | "ploj") to be interpreted as (verb (("ob object)|("ad" sentence)|"ploj")) and that's contrived. So to answer you: It can, and it does, but not always consistently. One can choose to be disappointed it's not perfect, but for just handing it a grammar and asking it to reason about it with no example text or chances to "practice" whatsoever, I think it's doing quite well. I've interviewed many software developers who'd struggle with a task like this. When prodded about the third example, it picks a more reasonable interpretation of the "|", though it's explanation starts out with an error, but then given a correct explanation: --- Q: How do you figure this? The full rule is "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). How do you figure the verb is allowed without being followed by either "ad" or "ob"? A: Upon closer examination, you are correct to point out that the structure following "sob" subject must indeed involve a verb according to the rule provided. The full rule: "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). indicates that the sentence must start with "sob", followed by a subject, and then either: a verb followed by "ob" and an object, a verb followed by "ad" and another sentence, or the standalone term "ploj". --- It goes on to give two correct alternatives to the erroneous one: --- * "sob foo ploj" — where "ploj" follows directly after the subject without a verb, or * "sob foo ab ob bar" — where "ab" is the verb followed by "ob" and "bar" as the object. --- While it's not perfect, it can very clearly "learn and reproduce multistep algorithms" to some extent. It also clearly needs to get better at it.

MeImCounting 3y ago

The one thing is that they seem to be using relatively small models. This may be a really damning result but I was under the impression that any generalization capabilities of LLMs appear in a non-linear fashion when you increase the parameter count to the tens of billions/trillions as in GPT4. It would be interesting if they could recreate the same experiment with a much larger model. Unfortunately I dont think thats likely to happen because of the resources required to train such models and the anti-open-source hysteria likely preventing larger models from being made publicly available much less the data they were trained on. Imagine that, stifling research and fearmongering reduces the usefulness of the science that does manage to get done.

mediaman 3y ago

There have been two criticisms of this paper floating around. 1. The test mechanism is to use prediction of sinusoidal series. While it's certainly possible to train transformers on mathematical functions, it's not clear why findings from a model trained on sinusoidal functions would generalize into the domain of written human language (which is ironic, given the paper's topic). 2. Even if it were true that these models don't generalize beyond their training, large LLMs' training corpus is basically all of written human knowledge. So then the goalpost has been moved to "well, they won't push the frontier of human knowledge forward," which seems to be a much diminished claim, since the vast majority of humans are also not pushing the frontier of human knowledge forward and instead use existing human knowledge to accomplish their daily goals.

hansvm 3y ago
You're using the idea of "all of human knowledge" differently in the two places it appears, and the gap in those definitions weakens the claim a bit. LLMs are trained on a tiny subset of the written human knowledge which we've proven to probably not be garbage and which is nicely formatted in simple text formats and which was published without too many paywalls on the web and on and on and on. It's a lot, and it definitely includes enough facts that no one person knows all the things the LLM "knows", but the average child knows plenty of things which never made it into that sort of a corpus. Yes, it's probably true that the vast majority of humans are also not pushing the frontier of human knowledge forward, but the vast majority of humans are working with a slightly different (partially overlapping) set of information than what the LLMs see.
- Isomorpheus 3y ago
  Can you name three things an average child knows which GPT-4 doesn't?
  - abeppu 3y ago
    How to drink from a cup without dribbling. The feeling of sunshine on one's skin. The experience of the passage of time.

pizza 3y ago

FWIW the paper title is focuses on quite a different conclusion than the submission title: Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models

lsh123 3y ago

Current AI models are approximation functions with huge number of parameters. These approximation functions are reasonably good at interpolation, meh at extrapolation, and have nothing to do with generalization.

famouswaffles 3y ago
Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485 "Supercharged Interpolation" is not a real thing. They generalize fine when the data incentivizes that.
- lsh123 3y ago
  You always can extrapolate. E.g. linear approximation for x^2 by 2 points will extrapolate reasonably well around these 2 points but will be bad with x -> +/- infinity. Similarly, there are examples where GPT invented legal cases when asked to create a legal brief.
  - samus 3y ago
    I asked GPT 3.5 how to access the Unihan character database[0] using the ICU4J library[1]. GPT 3.5 suggested to use the `com.ibm.icu.util.Unihan` class. It actually doesn't exist, but after pointing that out to it, GPT showed me how to parse the raw database files. [0]: https://www.unicode.org/charts/unihan.html https://www.unicode.org/charts/unihan.html [1]: https://unicode-org.github.io/icu/userguide/icu4j/ https://unicode-org.github.io/icu/userguide/icu4j/

Der_Einzige 3y ago

We humans don't even know when we are doing real extrapolation, and the vast majority of humans are interpolating. I bet many do nothing but interpolate their whole lives. So - and I say this as someone who writes NLP papers too - who cares?

datadrivenangel 3y ago

Overfitting much?

famouswaffles 3y ago

"Generalization" is always a data problem. If you trained it on one function class, of course that's all it learned to do. That's all it ever saw! If you want to learn arbitrary function classes to some degree, the solution is simple. Train it on many different function classes. Untrained models are as blank slate as you could possibly imagine. They're not even comparable to new born humans with millions of years of evolution baked in. The data you feed them is their world. Their only world.