4 ms·

Can we reverse engineer transformer models into human-understandable programs?

• darepublic 5y ago
  If they are human understandable then maybe they can be written that way in the first place?
  • ghgr 5y ago
    The papers states that human understandable and computationally efficient are often very opposed goals.
    • OJFord 5y ago
      Yes but in other areas we compile human understandable to computationally efficient.
• ludwigschubert 5y ago
  Still reading, but an early highlight for me is their rewriting of the attention mechanism in terms of tensor products. (Section “Attention Heads as Information Movement”) I always found the original formulation in terms of query/key/value/output matrices quite hard to operationally make sense of. It’s extremely exciting to see more “mechanistic interpretability”/“circuit analysis”-style work on ML architectures that are not convolutional neural networks!