sametoymak

3 ms·

1. 1.▲
   by sametoymak 3y ago

   IMO it is important to understand transformer mechanics through core ML themes like SVM and feature-selection. Our results are not interpretation, they are mathematically rigorous and numerically verifiable. That said, we have no intention
2. 2.▲
   by sametoymak 3y ago

   Here is what I meant: Standard SVM classifier: Maps an input sequence to a 0-1 label. Example: Take a paragraph and return its sentiment. During training, label is specified. Transformer's SVM: Takes input sequence, suppresses bad toke
3. 3.▲
   by sametoymak 3y ago

   This seems related to NTK literature i.e. wide neural nets behave like kernel regression. NTK is a great tool but a notable weakness is kernel view doesn't explain how the model learns new features. Transformer is also pretty different
4. 4.▲
   by sametoymak 3y ago

   This SVM summarizes the training dynamics of the attention layer, so there is no hidden-layer. It operates on the token embeddings of that layer. Essentially, weights of the attention layer converge (in direction) to the maximum margin sepa
5. 5.▲
   by sametoymak 3y ago

   I am one of the authors. The most critical aspect is that transformer is a "different kind of SVM". It solves an SVM that separates 'good' tokens within each input sequence from 'bad' tokens. This SVM serves as