4 ms·

This is why I'm very skeptical about the "Nobel prize level" claims. To win a Nobel prize you would have to produce something completely new. LLM will probably

This is why I'm very skeptical about the "Nobel prize level" claims. To win a Nobel prize you would have to produce something completely new. LLM will probably be able to reach a Ph.D. level of understanding existing research, but bringing something new is a different matter.

adamzwasserman 1y ago
LLMs do not understand anything. They have a very complex multidimensional "probability table" (more correctly a compressed geometric representation of token relationships) that they use to string together tokens (which have no semantic meaning), which then get converted to words that have semantic meaning to US, but not to the machine.
- KoolKat23 1y ago
  In abstract we do the exact same thing
  - tomfly 1y ago
    It’s hard to believe this when the llm “knows” so much more then us yet still can not be creative outside its training distribution
    - KoolKat23 1y ago
      When are we as humans creative outside our training data? It's very rare we actually discover something truly novel. This is often random, us stumbling onto it, brute force or purely by being at the right place at the right time. On the other hand, until it's proven it'd likely be considered a hallucination. You need to test something before you can dismiss it. (They did burn witches for discoveries back in the day, deemed witchcraft). We also reduce randomness and pre-train to avoid overfitting. Day to day human creative outputs as humans are actually less exciting when you think about it further, we build on pre-existing knowledge. No different to good prompt output with the right input. Humans are just more knowledgeable & smarter at the moment.
    - adamzwasserman 1y ago
      The LLM doesn't 'know' more than us - it has compressed more patterns from text than any human could process. That's not the same as knowledge. And yes, the training algorithms deliberately skew the distribution to maintain coherent output - without that bias toward seen patterns, it would generate nonsense. That's precisely why it can't be creative outside its training distribution: the architecture is designed to prevent novel combinations that deviate too far from learned patterns. Coherence and genuine creativity are in tension here
  - adamzwasserman 1y ago
    Perhaps in practice as well. It is well-established that our interaction with language far exceeds what we are conscious of.
    - KoolKat23 1y ago
      Absolutely, it is world model building.
- tomfly 1y ago
  Exactly. It’s been stated for a long time, before llms. For instance this paper https://home.csulb.edu/~cwallis/382/readings/482/searle.minds.brains.programs.bbs.1980.pdf https://home.csulb.edu/~cwallis/382/readings/482/searle.mind... Describes a translator who doesn’t know the language.
- DoctorOetker 1y ago
  Consider your human brain, and the full physical state, all the protons and neutrons some housed together in the same nucleus, some separate, together with all the electrons. Physics assigns probabilities to future states. Suppose you were in the middle of a conversation and about to express a next syllable (or token). That choice will depend on other choices ("what should I add next"), and further choices ("what is the best choice of words to express the thing I chose to express next etc. The probabilities are in principle calculable given a sufficiently detailed state. You are correct that LLM's correspond to a probability distribution (given you immediately corrected to say that this table is implicit and parametrized by a geometric token relationships.). But so does every expressor of language, humans included. The presence or absence of understanding can't be proven by mere association of with a "probability table", especially if such probability table is exactly expected from the perspective of physics, and if the models have continuously gained better and better performance by training them directly on human expressions!
KoolKat23 1y ago
Given a random prompt, the overall probability of seeing a specific output string is almost zero, since there are astronomically many possible token sequences. The same goes for humans. Most awards are built on novel research built on pre-existing works. This a LLM is capable of doing.
- adamzwasserman 1y ago
  LLMs don't use 'overall probability' in any meaningful sense. During training, gradient descent creates highly concentrated 'gravity wells' of correlated token relationships - the probability distribution is extremely non-uniform, heavily weighted toward patterns seen in training data. The model isn't selecting from 'astronomically many possible sequences' with equal probability; it's navigating pre-carved channels in high-dimensional space. That's fundamentally different from novel discovery.
  - KoolKat23 1y ago
    That's exactly the same for humans in the real world. You're focusing too close, abstract up a level. Your point relates to the "micro" system functioning, not the wider "macro" result (think emergent capabilities).
    - adamzwasserman 1y ago
      I'm afraid I'd need to see evidence before accepting that humans navigate 'pre-carved channels' in the same way LLMs do. Human learning involves direct interaction with physical reality, not just pattern matching on symbolic representations. Show me the equivalence or concede the point.
      - KoolKat23 1y ago
        Language and math are a world model of physical reality. You could not read a book and make sense of it if this were not true. An apple falls to the ground because of? gravity. In real life this is the answer, I'm very sure the pre-carved channel will also lead to gravity.

adamzwasserman 1y ago

LLMs do not understand anything. They have a very complex multidimensional "probability table" (more correctly a compressed geometric representation of token relationships) that they use to string together tokens (which have no semantic meaning), which then get converted to words that have semantic meaning to US, but not to the machine.

KoolKat23 1y ago
In abstract we do the exact same thing
- tomfly 1y ago
  It’s hard to believe this when the llm “knows” so much more then us yet still can not be creative outside its training distribution
  - KoolKat23 1y ago
    When are we as humans creative outside our training data? It's very rare we actually discover something truly novel. This is often random, us stumbling onto it, brute force or purely by being at the right place at the right time. On the other hand, until it's proven it'd likely be considered a hallucination. You need to test something before you can dismiss it. (They did burn witches for discoveries back in the day, deemed witchcraft). We also reduce randomness and pre-train to avoid overfitting. Day to day human creative outputs as humans are actually less exciting when you think about it further, we build on pre-existing knowledge. No different to good prompt output with the right input. Humans are just more knowledgeable & smarter at the moment.
  - adamzwasserman 1y ago
    The LLM doesn't 'know' more than us - it has compressed more patterns from text than any human could process. That's not the same as knowledge. And yes, the training algorithms deliberately skew the distribution to maintain coherent output - without that bias toward seen patterns, it would generate nonsense. That's precisely why it can't be creative outside its training distribution: the architecture is designed to prevent novel combinations that deviate too far from learned patterns. Coherence and genuine creativity are in tension here
- adamzwasserman 1y ago
  Perhaps in practice as well. It is well-established that our interaction with language far exceeds what we are conscious of.
  - KoolKat23 1y ago
    Absolutely, it is world model building.
tomfly 1y ago
Exactly. It’s been stated for a long time, before llms. For instance this paper https://home.csulb.edu/~cwallis/382/readings/482/searle.minds.brains.programs.bbs.1980.pdf https://home.csulb.edu/~cwallis/382/readings/482/searle.mind... Describes a translator who doesn’t know the language.
DoctorOetker 1y ago
Consider your human brain, and the full physical state, all the protons and neutrons some housed together in the same nucleus, some separate, together with all the electrons. Physics assigns probabilities to future states. Suppose you were in the middle of a conversation and about to express a next syllable (or token). That choice will depend on other choices ("what should I add next"), and further choices ("what is the best choice of words to express the thing I chose to express next etc. The probabilities are in principle calculable given a sufficiently detailed state. You are correct that LLM's correspond to a probability distribution (given you immediately corrected to say that this table is implicit and parametrized by a geometric token relationships.). But so does every expressor of language, humans included. The presence or absence of understanding can't be proven by mere association of with a "probability table", especially if such probability table is exactly expected from the perspective of physics, and if the models have continuously gained better and better performance by training them directly on human expressions!

KoolKat23 1y ago

Given a random prompt, the overall probability of seeing a specific output string is almost zero, since there are astronomically many possible token sequences. The same goes for humans. Most awards are built on novel research built on pre-existing works. This a LLM is capable of doing.

adamzwasserman 1y ago
LLMs don't use 'overall probability' in any meaningful sense. During training, gradient descent creates highly concentrated 'gravity wells' of correlated token relationships - the probability distribution is extremely non-uniform, heavily weighted toward patterns seen in training data. The model isn't selecting from 'astronomically many possible sequences' with equal probability; it's navigating pre-carved channels in high-dimensional space. That's fundamentally different from novel discovery.
- KoolKat23 1y ago
  That's exactly the same for humans in the real world. You're focusing too close, abstract up a level. Your point relates to the "micro" system functioning, not the wider "macro" result (think emergent capabilities).
  - adamzwasserman 1y ago
    I'm afraid I'd need to see evidence before accepting that humans navigate 'pre-carved channels' in the same way LLMs do. Human learning involves direct interaction with physical reality, not just pattern matching on symbolic representations. Show me the equivalence or concede the point.
    - KoolKat23 1y ago
      Language and math are a world model of physical reality. You could not read a book and make sense of it if this were not true. An apple falls to the ground because of? gravity. In real life this is the answer, I'm very sure the pre-carved channel will also lead to gravity.