3 ms·
I assumed that they had to train, otherwise how else would they get "inside" a transformer. I also feel a bit of bad smell from the article. Sounding revolutio
by dwa3592 8mo ago
I assumed that they had to train, otherwise how else would they get "inside" a transformer.
I also feel a bit of bad smell from the article. Sounding revolutionary with no details or clear explanation.
- D-Machine 8mo agoThere is no training in the usual sense of the term, i.e. no gradient descent, no differentiable loss function. They use deceptive language early on to make it sound this way, but near the end make it clear their model as is isn't actually differentiable, and in theory might still work if made differentiable. But they don't actually know. But IMO this is BS because I don't know how one would get or generate training data, or how one would define a continuous loss function that scores partially-correct / plausible outputs (e.g. is a "partially correct" program / algorithm / code even coherent, conceptually).
- YeGoblynQueenne 8mo agoYeah, a "100% correct" Sudoku solver fully trained by gradient descent from examples? That sure would be something entirely new. To answer dwa3592, it's always possible to set the weights of a neural net by hand, albeit extremely fiddly and normally only done "on paper". This is e.g. how the Turing-completeness of RNNs was shown back in the '90s: On the computational power of neural nets https://binds.cs.umass.edu/papers/1992_Siegelmann_COLT.pdf https://binds.cs.umass.edu/papers/1992_Siegelmann_COLT.pdf