4 ms·

If a new physics break through gets published today, no existing model will be able to fully integrate it - beyond a context window. If I put the paper in my se

If a new physics break through gets published today, no existing model will be able to fully integrate it - beyond a context window. If I put the paper in my session and it isnt in yours the model knows nothing. It wont retain it past that session. Models are trained, they do not learn.

sebastiennight 2mo ago
I think GP is using a different level of abstraction from yours in their metaphor. You are saying: Pre-Training == Everything you store in your memory throughout your life. Model weights == The lessons you learned Context == whatever you're currently thinking about One inference run == one thought They are saying: Pre-Training == building the DNA template of human brain through millions of years evolution. Model weights == Human DNA Context == Everything you store in your memory throughout your life, plus whatever you're currently thinking about One inference run == One human life. One instance == one human Applying their metaphor, your sentence becomes: > If a new physics breakthrough gets published today, no existing DNA structure will be able to fully integrate it - beyond an individual person. If I put the paper in my mind by learning it, and it isn't in yours, the DNA of human species stores nothing. It won't retain it past my lifetime. > The human species is trained (through evolution), it doesn't learn.
- orbital-decay 2mo ago
  The ladder in humans is even longer and wider than that, it's roughly: evolutionary pretraining of a complex molecular robot -> generational knowledge transfer and compression by the "parallelized agentic swarm" aka society -> individual lifetime learning due to neuroplasticity -> immediate attention (extremely narrow and volatile). Note how the individual is just one half of it.
  - sebastiennight 2mo ago
    Saying "half" relies on a lot of assumptions, as either side of that count can be made arbitrarily larger or smaller based on how many items you want to subdivide it into.

sebastiennight 2mo ago

I think GP is using a different level of abstraction from yours in their metaphor. You are saying: Pre-Training == Everything you store in your memory throughout your life. Model weights == The lessons you learned Context == whatever you're currently thinking about One inference run == one thought They are saying: Pre-Training == building the DNA template of human brain through millions of years evolution. Model weights == Human DNA Context == Everything you store in your memory throughout your life, plus whatever you're currently thinking about One inference run == One human life. One instance == one human Applying their metaphor, your sentence becomes: > If a new physics breakthrough gets published today, no existing DNA structure will be able to fully integrate it - beyond an individual person. If I put the paper in my mind by learning it, and it isn't in yours, the DNA of human species stores nothing. It won't retain it past my lifetime. > The human species is trained (through evolution), it doesn't learn.

orbital-decay 2mo ago
The ladder in humans is even longer and wider than that, it's roughly: evolutionary pretraining of a complex molecular robot -> generational knowledge transfer and compression by the "parallelized agentic swarm" aka society -> individual lifetime learning due to neuroplasticity -> immediate attention (extremely narrow and volatile). Note how the individual is just one half of it.
- sebastiennight 2mo ago
  Saying "half" relies on a lot of assumptions, as either side of that count can be made arbitrarily larger or smaller based on how many items you want to subdivide it into.