6 ms·

Thinking Like Transformers

161 points by ArtWomb 5y ago

PaulHoule 5y ago
A neural networks paper written as if computer science matters -- I love it!
failwhaleshark 5y ago
First, you have to make the shape-changing noises with your mouth while you turn into a Ford Focus. Then, you find some Energon. Next, red lasers.
0-_-0 5y ago
This is nice, however simple fully connected or convolutional networks can encode computer programs too. In fact, the hope for RNNs was that they will learn computer programs, but due to the weirdness of neural network training that never happened in practice to a satisfying degree. So finding a programming model for Transformers is not necessarily as practical as it first seems, since there's no guarantee that they will learn these programs from data.
- phreeza 5y ago
  I think the point here is that all architectures can execute a certain type of program. An RNN maps to a finite state automaton, and this is an attempt at showing what the natural class of programs a transformer can implement is.
  - siekmanj 5y ago
    Actually, I think it is conventional neural networks which can only approximate finite state machines. RNNs are (in theory, not so much in practice) Turing complete.
    - thomasahle 5y ago
      > RNNs are Turing complete. Turing completeness requires access to unlimited read/write memory. RNNs only have a fixed dimensional state. I guess I'm theory that starte is continuous, but it has to be a pretty optimistic model that assumes we can handle unbounded data like that.
      - deleted 5y ago
        [deleted]
      - peytn 5y ago
        Intuitively I think as long as your activation function is sufficiently expressive (e.g. not a step function or something), you should be go to go in theory since that’s what you’re feeding back. Might take a while.
        thomasahle 5y ago
        The word I was looking for is "robust". Any realistic model must be able to accept tiny perturbations (like Gaussian noise) of the vectors, since that's how floating arithmetic works. An RNN can't be robustly Turing complete.
G3rn0ti 5y ago
Oh, for a moment I thought the article was about Autobots …
- jhgb 5y ago
  It's actually about Decepticons?
  - failwhaleshark 5y ago
    I am... Megatron!
    - lapp0 5y ago
      Inferior fleshlings lay before me... as barely breathing hunks of meat With the blood of Unicron in my veins, I reign like a god A god amongst insects I have existed from the morning of the universe And I shall exist until the last star falls from the night My ultimate peace would be granted by the destruction of all life, stars and nebulae Leaving only nothingness and void Although I have taken the form of this machine I am all men as I am no man And therefore I am a god Even in death there is no command but mine Your race is of no consequence Laserbeak... Kill them all
- teekert 5y ago
  Autobots, transform and roll out! (I share your disappointment fwiw)
- trm42 5y ago
  Same here! Was expecting speculative implementation or comparison how their setup would be and what kind of algorithms etc there could be.
- libria 5y ago
  "As an Autobot, should I buy life insurance or car insurance?"
  - 5y ago
baxinho0312 5y ago
As someone working in EE on power transformers, I am deeply disappointed that this article is about neural networks....
- gk1 5y ago
  Dang, please consider saying something in the guidelines about these kinds of comments?
  - mchusma 5y ago
    As a counterpoint, I prefer a world where comments are not overly policed. I believe it stifles creativity, and I think comments like yours harm not help the community by making people less comfortable sharing.
    - failwhaleshark 5y ago
      Exactly. The culture of "no" and control-freaks trying to shackle others into thinking down a linear path. This often leads to ideological homogeneity and pushes a large fraction of people away, amplifying homogeneity. Instead of downvoting or flagging for merely disinterest or disagreement, perhaps there should be some sort of helpful "hide button" in the form of a plus-to-minus sign next to "parent"?
      - meowkit 5y ago
        But then how will I attempt to gain feelings of control, and thus perceived power, during my life in the salient face of my inevitable mortal demise? /s https://en.m.wikipedia.org/wiki/Terror_management_theory https://en.m.wikipedia.org/wiki/Terror_management_theory
        failwhaleshark 5y ago
        Isn't that why most people become police officers and other authority figures? You'll just have to torture living things, wet the bed, and set fires like the rest of us. /s /s /s /s /s Don't forget moral panicking, outrage crybullying, serial scapegoats crucifixion, taking-out aggression, bikeshedding, and cyberdisinhibitionism are also part of this complete breakfast.
xiphias2 5y ago
I love this paper, I always had an issue of visualizing transformers from reading ML papers, with this I can just play with it. These are simple higher-order functions that can be implemented in any language though, so porting to Python and playing with it in a Jupyter notebook is trivial.
- anentropic 5y ago
  I'm not sure if it's from the authors of the paper, but this appears to be that: https://github.com/tech-srl/RASP https://github.com/tech-srl/RASP
  - polynomial 5y ago
    The paper's lead author is the sole committer on that repo.
xmaayy 5y ago
(DID NOT READ THE ENTIRE PAPER, only the abstract, the definition of the language and some of the experiments) Note sure how useful this is in the larger context of transformers. Transformers (and deep networks in general) are often used when the logic to be used in solving a problem is largely unknown. Example -- How do you write a RASP program that identifies names in a document? They do have some simple RASP examples in the paper of things that a transformer model can accomplish (Symbolic Reasoning in Transformers) but, again, this is usually something that the model can do as a result of the task it was originally trained for, not a task in and of itself.
- canjobear 5y ago
  The point isn't that you can write programs in RASP. It's that you can use RASP as a tool to reason about what tasks Transformers will be good and bad at and how their architecture influences that.
KhoomeiK 5y ago
I believe [1] is the work they're mainly referencing for their RNN equivalent/inspiration. [1] https://nlp.stanford.edu/~johnhew/rnns-hierarchy.html https://nlp.stanford.edu/~johnhew/rnns-hierarchy.html
neogodless 5y ago
To go a bit meta, I think this link contributes to Hacker News because it sparks curiosity. While I was among those wondering if this would look into some kind of computational model that did what Transformers do (at least in the modern movies), scan an object in order to reconstruct it / reconfigure itself into that object, I quickly realized this is about something else. But it is something interesting and worth learning more about[0]. So just get over your prime instinct to be disappointed that the subject doesn't match your expectations, and change your attitude towards information shared that feels alien. [0] https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) https://en.wikipedia.org/wiki/Transformer_(machine_learning_...
- mch82 5y ago
  They also did this in the original cartoon… actually, their spaceship scanned earth & programmed them with earth-like vehicle forms.
lincpa 5y ago
Another robots and transformers on 2021-02-02: https://github.com/linpengcheng/PurefunctionPipelineDataflow#Robots-and-transformers https://github.com/linpengcheng/PurefunctionPipelineDataflow...

161 points by ArtWomb 5y ago

A neural networks paper written as if computer science matters -- I love it!

First, you have to make the shape-changing noises with your mouth while you turn into a Ford Focus. Then, you find some Energon. Next, red lasers.

0-_-0 5y ago

This is nice, however simple fully connected or convolutional networks can encode computer programs too. In fact, the hope for RNNs was that they will learn computer programs, but due to the weirdness of neural network training that never happened in practice to a satisfying degree. So finding a programming model for Transformers is not necessarily as practical as it first seems, since there's no guarantee that they will learn these programs from data.

phreeza 5y ago
I think the point here is that all architectures can execute a certain type of program. An RNN maps to a finite state automaton, and this is an attempt at showing what the natural class of programs a transformer can implement is.
- siekmanj 5y ago
  Actually, I think it is conventional neural networks which can only approximate finite state machines. RNNs are (in theory, not so much in practice) Turing complete.
  - thomasahle 5y ago
    > RNNs are Turing complete. Turing completeness requires access to unlimited read/write memory. RNNs only have a fixed dimensional state. I guess I'm theory that starte is continuous, but it has to be a pretty optimistic model that assumes we can handle unbounded data like that.
    - deleted 5y ago
      [deleted]
    - peytn 5y ago
      Intuitively I think as long as your activation function is sufficiently expressive (e.g. not a step function or something), you should be go to go in theory since that’s what you’re feeding back. Might take a while.
      - thomasahle 5y ago
        The word I was looking for is "robust". Any realistic model must be able to accept tiny perturbations (like Gaussian noise) of the vectors, since that's how floating arithmetic works. An RNN can't be robustly Turing complete.

G3rn0ti 5y ago

Oh, for a moment I thought the article was about Autobots …

jhgb 5y ago
It's actually about Decepticons?
- failwhaleshark 5y ago
  I am... Megatron!
  - lapp0 5y ago
    Inferior fleshlings lay before me... as barely breathing hunks of meat With the blood of Unicron in my veins, I reign like a god A god amongst insects I have existed from the morning of the universe And I shall exist until the last star falls from the night My ultimate peace would be granted by the destruction of all life, stars and nebulae Leaving only nothingness and void Although I have taken the form of this machine I am all men as I am no man And therefore I am a god Even in death there is no command but mine Your race is of no consequence Laserbeak... Kill them all
teekert 5y ago
Autobots, transform and roll out! (I share your disappointment fwiw)
trm42 5y ago
Same here! Was expecting speculative implementation or comparison how their setup would be and what kind of algorithms etc there could be.
libria 5y ago
"As an Autobot, should I buy life insurance or car insurance?"
- 5y ago

baxinho0312 5y ago

As someone working in EE on power transformers, I am deeply disappointed that this article is about neural networks....

gk1 5y ago
Dang, please consider saying something in the guidelines about these kinds of comments?
- mchusma 5y ago
  As a counterpoint, I prefer a world where comments are not overly policed. I believe it stifles creativity, and I think comments like yours harm not help the community by making people less comfortable sharing.
  - failwhaleshark 5y ago
    Exactly. The culture of "no" and control-freaks trying to shackle others into thinking down a linear path. This often leads to ideological homogeneity and pushes a large fraction of people away, amplifying homogeneity. Instead of downvoting or flagging for merely disinterest or disagreement, perhaps there should be some sort of helpful "hide button" in the form of a plus-to-minus sign next to "parent"?
    - meowkit 5y ago
      But then how will I attempt to gain feelings of control, and thus perceived power, during my life in the salient face of my inevitable mortal demise? /s https://en.m.wikipedia.org/wiki/Terror_management_theory https://en.m.wikipedia.org/wiki/Terror_management_theory
      - failwhaleshark 5y ago
        Isn't that why most people become police officers and other authority figures? You'll just have to torture living things, wet the bed, and set fires like the rest of us. /s /s /s /s /s Don't forget moral panicking, outrage crybullying, serial scapegoats crucifixion, taking-out aggression, bikeshedding, and cyberdisinhibitionism are also part of this complete breakfast.

xiphias2 5y ago

I love this paper, I always had an issue of visualizing transformers from reading ML papers, with this I can just play with it. These are simple higher-order functions that can be implemented in any language though, so porting to Python and playing with it in a Jupyter notebook is trivial.

anentropic 5y ago
I'm not sure if it's from the authors of the paper, but this appears to be that: https://github.com/tech-srl/RASP https://github.com/tech-srl/RASP
- polynomial 5y ago
  The paper's lead author is the sole committer on that repo.

xmaayy 5y ago

(DID NOT READ THE ENTIRE PAPER, only the abstract, the definition of the language and some of the experiments) Note sure how useful this is in the larger context of transformers. Transformers (and deep networks in general) are often used when the logic to be used in solving a problem is largely unknown. Example -- How do you write a RASP program that identifies names in a document? They do have some simple RASP examples in the paper of things that a transformer model can accomplish (Symbolic Reasoning in Transformers) but, again, this is usually something that the model can do as a result of the task it was originally trained for, not a task in and of itself.

canjobear 5y ago
The point isn't that you can write programs in RASP. It's that you can use RASP as a tool to reason about what tasks Transformers will be good and bad at and how their architecture influences that.

KhoomeiK 5y ago

I believe [1] is the work they're mainly referencing for their RNN equivalent/inspiration. [1] https://nlp.stanford.edu/~johnhew/rnns-hierarchy.html https://nlp.stanford.edu/~johnhew/rnns-hierarchy.html

neogodless 5y ago

To go a bit meta, I think this link contributes to Hacker News because it sparks curiosity. While I was among those wondering if this would look into some kind of computational model that did what Transformers do (at least in the modern movies), scan an object in order to reconstruct it / reconfigure itself into that object, I quickly realized this is about something else. But it is something interesting and worth learning more about[0]. So just get over your prime instinct to be disappointed that the subject doesn't match your expectations, and change your attitude towards information shared that feels alien. [0] https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) https://en.wikipedia.org/wiki/Transformer_(machine_learning_...

mch82 5y ago
They also did this in the original cartoon… actually, their spaceship scanned earth & programmed them with earth-like vehicle forms.

lincpa 5y ago

Another robots and transformers on 2021-02-02: https://github.com/linpengcheng/PurefunctionPipelineDataflow#Robots-and-transformers https://github.com/linpengcheng/PurefunctionPipelineDataflow...