3 ms·

Surely learning an algorithm is memorization? I still remember the physics equations from when I was at school (a long time ago) because it was drilled into us

Surely learning an algorithm is memorization? I still remember the physics equations from when I was at school (a long time ago) because it was drilled into us repeatedly. Stuff like v = u + at, or s=tu + 1/2 (at)^2, etc etc. I don't really recall how they were derived etc but I did memorize them. It definitely doesn't mean I'm good at physics or understand the deeper meaning of these equations. I also had to memorize bubble sort, merge, heap, quicksort etc when I was in university, but I don't think I could invent these sorts from first principles without looking it up. Memorization doesn't really equal understanding, it's just memorization.

sushibowl 3y ago
> Surely learning an algorithm is memorization? Not in this context, no. An LLM is never given any algorithm to memorise. It is given only input->output sets, and it "learns" what the algorithm is from those sets. We know that it does this because it is able to generalize that algorithm to inputs and outputs outside of its set of training examples. So we know that it doesn't only memorise which input connects to which output, and regurgitate that information. It has come to "understand" the formula that connects the input to the output, without ever being given that formula.
- quantum_state 3y ago
  It’s fitting a function to data and guessing the value of the function for a new input. It does not know what is under the hood of the function, etc., and see the implications when the function produces a wrong value … etc.
  - astrange 3y ago
    It can do that. Verifying an answer is just another algorithm it can learn. LLMs mostly can't do math but that, like most of their other flaws, is because of the tokenizer.
retrac 3y ago
> Surely learning an algorithm is memorization? Yes. In a sense, every algorithm can be reduced to memorization, where you precalculate all possible inputs to all possible outputs and store them in a giant lookup table.

sushibowl 3y ago

> Surely learning an algorithm is memorization? Not in this context, no. An LLM is never given any algorithm to memorise. It is given only input->output sets, and it "learns" what the algorithm is from those sets. We know that it does this because it is able to generalize that algorithm to inputs and outputs outside of its set of training examples. So we know that it doesn't only memorise which input connects to which output, and regurgitate that information. It has come to "understand" the formula that connects the input to the output, without ever being given that formula.

quantum_state 3y ago
It’s fitting a function to data and guessing the value of the function for a new input. It does not know what is under the hood of the function, etc., and see the implications when the function produces a wrong value … etc.
- astrange 3y ago
  It can do that. Verifying an answer is just another algorithm it can learn. LLMs mostly can't do math but that, like most of their other flaws, is because of the tokenizer.

retrac 3y ago

> Surely learning an algorithm is memorization? Yes. In a sense, every algorithm can be reduced to memorization, where you precalculate all possible inputs to all possible outputs and store them in a giant lookup table.