4 ms·

So, what I'm trying to understand, and I can't find any clear information about that in the article, is how they "compiled" e.g. the Sudoku solver into a Transf

So, what I'm trying to understand, and I can't find any clear information about that in the article, is how they "compiled" e.g. the Sudoku solver into a Transformer's weights. Did they do it manually? Say, they took the source of a hand-coded Sudoku solver and put it through their code-to-weight compiler, and thus compiled the code to the Transformer weights? Or did they go the Good, Old-Fashioned, Deep Learning way and train their Transformer to learn a ("100% correct"!) Sudoku solver from examples? And, if the latter, where's the details of the training? What did they train with? What did they train on? How did they train? etc etc. Very light on details that article is.

MadnessASAP 8mo ago
My interpretation is that they built a simple virtual machine directly into the weights, then compiled a WASM runtime for that machine, then compiled the solver to that runtime.
- YeGoblynQueenne 8mo ago
  That's more or less what I got, also, but it's hard to tell. What a very annoying article, in its vagueness.
gavinray 8mo ago
The article states they trained a WASM interpreter and programs are represented as WASM bytecode
- D-Machine 8mo ago
  Nope, they encoded or compiled in a simple VM / WASM interpreter to the transformer weights, there is no training. You'd be forgiven for this misreading, as they deliberately mislead early on that their model is (in principle) trainable, but later admit that their actual model is not actually differentiable, but that a differentiable approximation "should" still work (despite no info about what loss function or training data could allow scoring partially correct / incomplete program outputs).
  - YeGoblynQueenne 8mo ago
    Thanks, but where do they say that? I can only find this instance of "different" (as in "differentiable") in their article: Because the execution trace is part of the forward pass, the whole process remains differentiable: we can even propagate gradients through the computation itself. That makes this fundamentally different from an external tool. It becomes a trainable computational substrate that can be integrated directly into a larger model.
    - D-Machine 7mo ago
      In the section "Programs into weights & training beyond gradient descent", near the end, they say: [...] *the compilation machinery we built for generating those weights** can go further. In principle, arbitrary programs can be compiled directly into the transformer weights, bypassing the need to represent them as token sequences at all. [...] [my emphasis] In the same section, they also continue: Weights become a deployment target: instead of learning software-like behavior, models contain compiled program logic. If logic can be compiled into weights, then gradient descent is no longer the only way to modify a model. Weight compilation provides another route for inserting structure, algorithms, and guarantees directly into a network. So they (almost-invisibly) admit they compile in the weights, but make it clearer this was the whole intention the whole time in later sentences.

MadnessASAP 8mo ago

My interpretation is that they built a simple virtual machine directly into the weights, then compiled a WASM runtime for that machine, then compiled the solver to that runtime.

YeGoblynQueenne 8mo ago
That's more or less what I got, also, but it's hard to tell. What a very annoying article, in its vagueness.

gavinray 8mo ago

The article states they trained a WASM interpreter and programs are represented as WASM bytecode

D-Machine 8mo ago
Nope, they encoded or compiled in a simple VM / WASM interpreter to the transformer weights, there is no training. You'd be forgiven for this misreading, as they deliberately mislead early on that their model is (in principle) trainable, but later admit that their actual model is not actually differentiable, but that a differentiable approximation "should" still work (despite no info about what loss function or training data could allow scoring partially correct / incomplete program outputs).
- YeGoblynQueenne 8mo ago
  Thanks, but where do they say that? I can only find this instance of "different" (as in "differentiable") in their article: Because the execution trace is part of the forward pass, the whole process remains differentiable: we can even propagate gradients through the computation itself. That makes this fundamentally different from an external tool. It becomes a trainable computational substrate that can be integrated directly into a larger model.
  - D-Machine 7mo ago
    In the section "Programs into weights & training beyond gradient descent", near the end, they say: [...] *the compilation machinery we built for generating those weights** can go further. In principle, arbitrary programs can be compiled directly into the transformer weights, bypassing the need to represent them as token sequences at all. [...] [my emphasis] In the same section, they also continue: Weights become a deployment target: instead of learning software-like behavior, models contain compiled program logic. If logic can be compiled into weights, then gradient descent is no longer the only way to modify a model. Weight compilation provides another route for inserting structure, algorithms, and guarantees directly into a network. So they (almost-invisibly) admit they compile in the weights, but make it clearer this was the whole intention the whole time in later sentences.