3 ms·
A related line of work is "Thinking Like Transformers" [1]. They introduce a primitive programming language, RASP, which is composed of operations capable of be
by teddykoker 3y ago
A related line of work is "Thinking Like Transformers" [1]. They introduce a primitive programming language, RASP, which is composed of operations capable of being modeled with transformer components, and demonstrate how different programs can be written with it, e.g. histograms, sorting. Sasha Rush and Gail Weiss have an excellent blog post on it as well [2]. Follow on work actually demonstrated how RASP-like programs could actually be compiled into model weights without training [3].
[1] https://arxiv.org/abs/2106.06981 https://arxiv.org/abs/2106.06981
[2] https://srush.github.io/raspy/ https://srush.github.io/raspy/
[3] https://arxiv.org/abs/2301.05062 https://arxiv.org/abs/2301.05062
- newhouseb 3y agoHuge fan of RASP et al. If you enjoy this space, might be fun to take a glance at some of my work on HandCrafted Transformers [1] wherein I hand-pick the weights in a transformer model to do long-handed addition similar to how humans learn to do it in gradeshcool. [1] https://colab.research.google.com/github/newhouseb/handcrafted/blob/main/HandCrafted.ipynb https://colab.research.google.com/github/newhouseb/handcraft...
- theGnuMe 3y agoIt seems like a functional language like Haskell would be the right tool for this. Also going from a net to code would be super interesting in terms of explain-ability.