3 ms·

if you understood the article, please correct my understanding - they created a new training dataset which also has computation solving step by step (multiplyi

by dwa3592 8mo ago

if you understood the article, please correct my understanding - they created a new training dataset which also has computation solving step by step (multiplying two numbers or playing sudoku) and then trained a transformer on it- as a result, the model performs the computation(multiplying two numbers) "inside" itself instead of calling calculator (or python)? ++ And they also figured out how to make attention faster?

YeGoblynQueenne 8mo ago
I can't see anything about "training a transformer". I'm trying to understand if e.g. the Sudoku solver was learned from examples (in which case, what examples?) or whether it was manually coded and then "compiled" into weights.
- dwa3592 8mo ago
  I assumed that they had to train, otherwise how else would they get "inside" a transformer. I also feel a bit of bad smell from the article. Sounding revolutionary with no details or clear explanation.
  - D-Machine 8mo ago
    There is no training in the usual sense of the term, i.e. no gradient descent, no differentiable loss function. They use deceptive language early on to make it sound this way, but near the end make it clear their model as is isn't actually differentiable, and in theory might still work if made differentiable. But they don't actually know. But IMO this is BS because I don't know how one would get or generate training data, or how one would define a continuous loss function that scores partially-correct / plausible outputs (e.g. is a "partially correct" program / algorithm / code even coherent, conceptually).
    - YeGoblynQueenne 8mo ago
      Yeah, a "100% correct" Sudoku solver fully trained by gradient descent from examples? That sure would be something entirely new. To answer dwa3592, it's always possible to set the weights of a neural net by hand, albeit extremely fiddly and normally only done "on paper". This is e.g. how the Turing-completeness of RNNs was shown back in the '90s: On the computational power of neural nets https://binds.cs.umass.edu/papers/1992_Siegelmann_COLT.pdf https://binds.cs.umass.edu/papers/1992_Siegelmann_COLT.pdf
- jamilton 8mo ago
  I would assume it was manually coded.

by dwa3592 8mo ago

YeGoblynQueenne 8mo ago

I can't see anything about "training a transformer". I'm trying to understand if e.g. the Sudoku solver was learned from examples (in which case, what examples?) or whether it was manually coded and then "compiled" into weights.

dwa3592 8mo ago
I assumed that they had to train, otherwise how else would they get "inside" a transformer. I also feel a bit of bad smell from the article. Sounding revolutionary with no details or clear explanation.
- D-Machine 8mo ago
  There is no training in the usual sense of the term, i.e. no gradient descent, no differentiable loss function. They use deceptive language early on to make it sound this way, but near the end make it clear their model as is isn't actually differentiable, and in theory might still work if made differentiable. But they don't actually know. But IMO this is BS because I don't know how one would get or generate training data, or how one would define a continuous loss function that scores partially-correct / plausible outputs (e.g. is a "partially correct" program / algorithm / code even coherent, conceptually).
  - YeGoblynQueenne 8mo ago
    Yeah, a "100% correct" Sudoku solver fully trained by gradient descent from examples? That sure would be something entirely new. To answer dwa3592, it's always possible to set the weights of a neural net by hand, albeit extremely fiddly and normally only done "on paper". This is e.g. how the Turing-completeness of RNNs was shown back in the '90s: On the computational power of neural nets https://binds.cs.umass.edu/papers/1992_Siegelmann_COLT.pdf https://binds.cs.umass.edu/papers/1992_Siegelmann_COLT.pdf
jamilton 8mo ago
I would assume it was manually coded.