3 ms·

Statistical models generalize. If you train a model that f(x) = 5 and f(x+1) = 6, the number 7 doesn't have to exist in the training data for the model to give

Statistical models generalize. If you train a model that f(x) = 5 and f(x+1) = 6, the number 7 doesn't have to exist in the training data for the model to give you a correct answer for f(x+2) Similarly, if there are millions of academic papers and thousands of peer reviews in the training data, a review of this exact paper doesn't need to be in there for the LLM to write something convincing. (I say "convincing" rather than "correct" since, the author himself admits that he doesn't agree with all the LLM's comments.) I tend to recommend people learn these things from first principles (e.g. build a small neural network, explore deep learning, build a language model) to gain a better intuition. There's really no "magic" at work here.

selridge 7mo ago
Ok cool cool. Instead of pretending you need to teach me, you could engage with what I'm saying or even the OP! "I don't know how you get here from "predict the next word"" is not really so much a statement of ignorance where someone needs you to step in but a reflection that perhaps the tech is not so easily explained as that. No magic needs to be present for that to be the case.
- wavemode 7mo ago
  If you disagree with someone on the internet, you can just say "I disagree, and here's why". You don't have to aggressively accuse them of "not engaging" with the text. I engaged. You just don't like what I wrote. That's okay.
  - selridge 7mo ago
    Thanks but no thanks.
c22 7mo ago
> If you train a model that f(x) = 5 and f(x+1) = 6, the number 7 doesn't have to exist in the training data for the model to give you a correct answer for f(x+2) This is an interesting claim to me. Are there any models that exist that have been trained with a (single digit) number omitted from the training data? If such a model does exist, how does it represent the answer? (What symbol does it use for the '7'?)
- wavemode 7mo ago
  When I say "model" here I'm referring to any statistical model (in this example, probably linear regression). Not specifically large language models / neural networks.
  - c22 7mo ago
    Gotcha, I don't think I know enough about it. What constitutes training data for a for a (non neural network) statistical model? Is this something I could play around with myself with pen and paper?
    - heavyset_go 7mo ago
      You can write an f(x) and record the input and output and that can be your training data. Or just download some time-series data or something.
    - nairboon 7mo ago
      Just the raw numbers? You list the y's and the x's and the model is approximating y=f(x) from the above example. You can totally do it with pen and paper. This is what it'd look like (for linear regression): https://observablehq.com/@yizhe-ang/interactive-visualization-of-linear-regression https://observablehq.com/@yizhe-ang/interactive-visualizatio...
red75prime 7mo ago
I think the relevant question is: can a statistical model (or a transformer, in particular) generalize to general reasoning ability?
- deleted 7mo ago
  [deleted]
kristiandupont 7mo ago
I had Claude help me get a program written for Linux to compile on macOS. The program is written in a programming language the author invented for the project, a pretty unusual one (for example, it allows spaces in variable names). Claude figured out how the language worked and debugged segfaults until the compiler compiled, and then until the program did. That might not be magic, but it shows a level of sophistication where referring to “statistics” is about as meaningful as describing a person as the statistics of electrical impulses between neurons.
- compass_copium 7mo ago
  But the programming language has explicitly laid out rules. It was not trained on those sets of rules, but it was trained on many trillions of lines of code. It has a map of how programs work, and an explanation of this new language. It's using training data and data it's fed to generate that result.
  - selridge 7mo ago
    What doesn't that explain tho? What behavior would you need to see for that explanation to no longer hold? Because it seems like it explains too much.
    - deleted 7mo ago
      [deleted]
    - BobaFloutist 7mo ago
      I don't know how you'd prompt this, but if there was a clean example of an A.I. coming up with an idea that's completely novel in more than details, it would be compelling evidence that these next-token predictors have some weird emergent properties that don't necessarily follow from intricate, sophisticated webs of token-prediction. E.g. "What might be a room-temperature superconductor" -> "some plausible iteration on existing high-temperature superconductors based on our current understanding of the underlying physics" would not be outside how we currently understand them. "What might be a room-temperature superconductor?" -> "some completely outlandish material that nobody has studied before and, when examined, seems to have higher temperature superconducting than we would predict" would provoke some serious questions. A fun experiment I've heard suggested is training a model on all scientific understanding just up to some counterintuitive quantum leap in scientific understanding, say, Einstein's theory of relativity, and then seeing if you can prompt it to "discover" or "invent" said leap, without explicitly telling it what to look for. This would of course be pretty hard to prove, but if you could get it to work on a local model, publish the training set and parameters so that anyone can replicate it on their own machine, that could be pretty darn compelling.
Kim_Bruning 7mo ago
If you run an LLM in an autoregressive loop you can get it to emulate a turing machine though. That sort of changes the complexity class of the system just a touch. 'Just predicts the next word' hits different when the loop is doing general computation. Took me a bit of messing around, but try to write out each state sequentially, with a check step between each.
arkh 7mo ago
I expected (and still expect) a lot from LLM with cross disciplinary research. I think they should be the perfect tool to find methods or results in a field which look like it could be used in another field.
- WithinReason 7mo ago
  This might actually be a limitation of the "predict next word" approach since the network is never trained to predict a result in one field from a result in another. It might still make the connection though, but not as easily.
ainch 7mo ago
Sorry but this is famously not true! There is no guarantee that statistical models generalise. In your example, whether or not your model generalises depends entirely on what f(x) you use - depending on the complexity of your function class f(x+2) could be 7, 8, or -500. One of the surprises of deep learning is that it can, sometimes, defy prior statistical learning theory to generalise, but this is still poorly understood. Concepts like grokking, double descent, and the implicit bias of gradient descent are driving a lot of new research into the underlying dynamics of deep learning. But I'd say it is pretty ahistoric to claim that this is obvious or trivial - decades of work studied "overfitting" and related problems where statistical models fail to generalise or even interpolate within the support of their training data.

selridge 7mo ago

Ok cool cool. Instead of pretending you need to teach me, you could engage with what I'm saying or even the OP! "I don't know how you get here from "predict the next word"" is not really so much a statement of ignorance where someone needs you to step in but a reflection that perhaps the tech is not so easily explained as that. No magic needs to be present for that to be the case.

wavemode 7mo ago
If you disagree with someone on the internet, you can just say "I disagree, and here's why". You don't have to aggressively accuse them of "not engaging" with the text. I engaged. You just don't like what I wrote. That's okay.
- selridge 7mo ago
  Thanks but no thanks.

c22 7mo ago

> If you train a model that f(x) = 5 and f(x+1) = 6, the number 7 doesn't have to exist in the training data for the model to give you a correct answer for f(x+2) This is an interesting claim to me. Are there any models that exist that have been trained with a (single digit) number omitted from the training data? If such a model does exist, how does it represent the answer? (What symbol does it use for the '7'?)

wavemode 7mo ago
When I say "model" here I'm referring to any statistical model (in this example, probably linear regression). Not specifically large language models / neural networks.
- c22 7mo ago
  Gotcha, I don't think I know enough about it. What constitutes training data for a for a (non neural network) statistical model? Is this something I could play around with myself with pen and paper?
  - heavyset_go 7mo ago
    You can write an f(x) and record the input and output and that can be your training data. Or just download some time-series data or something.
  - nairboon 7mo ago
    Just the raw numbers? You list the y's and the x's and the model is approximating y=f(x) from the above example. You can totally do it with pen and paper. This is what it'd look like (for linear regression): https://observablehq.com/@yizhe-ang/interactive-visualization-of-linear-regression https://observablehq.com/@yizhe-ang/interactive-visualizatio...

red75prime 7mo ago

I think the relevant question is: can a statistical model (or a transformer, in particular) generalize to general reasoning ability?

deleted 7mo ago
[deleted]

kristiandupont 7mo ago

I had Claude help me get a program written for Linux to compile on macOS. The program is written in a programming language the author invented for the project, a pretty unusual one (for example, it allows spaces in variable names). Claude figured out how the language worked and debugged segfaults until the compiler compiled, and then until the program did. That might not be magic, but it shows a level of sophistication where referring to “statistics” is about as meaningful as describing a person as the statistics of electrical impulses between neurons.

compass_copium 7mo ago
But the programming language has explicitly laid out rules. It was not trained on those sets of rules, but it was trained on many trillions of lines of code. It has a map of how programs work, and an explanation of this new language. It's using training data and data it's fed to generate that result.
- selridge 7mo ago
  What doesn't that explain tho? What behavior would you need to see for that explanation to no longer hold? Because it seems like it explains too much.
  - deleted 7mo ago
    [deleted]
  - BobaFloutist 7mo ago
    I don't know how you'd prompt this, but if there was a clean example of an A.I. coming up with an idea that's completely novel in more than details, it would be compelling evidence that these next-token predictors have some weird emergent properties that don't necessarily follow from intricate, sophisticated webs of token-prediction. E.g. "What might be a room-temperature superconductor" -> "some plausible iteration on existing high-temperature superconductors based on our current understanding of the underlying physics" would not be outside how we currently understand them. "What might be a room-temperature superconductor?" -> "some completely outlandish material that nobody has studied before and, when examined, seems to have higher temperature superconducting than we would predict" would provoke some serious questions. A fun experiment I've heard suggested is training a model on all scientific understanding just up to some counterintuitive quantum leap in scientific understanding, say, Einstein's theory of relativity, and then seeing if you can prompt it to "discover" or "invent" said leap, without explicitly telling it what to look for. This would of course be pretty hard to prove, but if you could get it to work on a local model, publish the training set and parameters so that anyone can replicate it on their own machine, that could be pretty darn compelling.

Kim_Bruning 7mo ago

If you run an LLM in an autoregressive loop you can get it to emulate a turing machine though. That sort of changes the complexity class of the system just a touch. 'Just predicts the next word' hits different when the loop is doing general computation. Took me a bit of messing around, but try to write out each state sequentially, with a check step between each.

arkh 7mo ago

I expected (and still expect) a lot from LLM with cross disciplinary research. I think they should be the perfect tool to find methods or results in a field which look like it could be used in another field.

WithinReason 7mo ago
This might actually be a limitation of the "predict next word" approach since the network is never trained to predict a result in one field from a result in another. It might still make the connection though, but not as easily.

ainch 7mo ago

Sorry but this is famously not true! There is no guarantee that statistical models generalise. In your example, whether or not your model generalises depends entirely on what f(x) you use - depending on the complexity of your function class f(x+2) could be 7, 8, or -500. One of the surprises of deep learning is that it can, sometimes, defy prior statistical learning theory to generalise, but this is still poorly understood. Concepts like grokking, double descent, and the implicit bias of gradient descent are driving a lot of new research into the underlying dynamics of deep learning. But I'd say it is pretty ahistoric to claim that this is obvious or trivial - decades of work studied "overfitting" and related problems where statistical models fail to generalise or even interpolate within the support of their training data.