3 ms·

Explain like I’m a first year CS major?

• thesehands 5y ago
  Transformers suffer from a quadratic bottleneck when calculating attention. Much work has been done investigating where memory can be saved by being more explicit on which attentions to calculate. This repo implements transformers with noted improvements
• erik_seaberg 5y ago
  It’s a kind of machine learning model: https://medium.com/inside-machine-learning/what-is-a-transformer-d07dd1fbec04 https://medium.com/inside-machine-learning/what-is-a-transfo...
• ericjang 5y ago
  vanilla neural networks from the 1950's look like def f(x): for _ in range(3): x = g(Wx + b) return x Essentially it is a matrix multiplication, a vector addition, a non-linearity. Transformers are a modification to that architecture - using different multiplications, additions, and non-linearities. both of these are general in the sense that if you have enough of them, they can approximate any function. The ones used for transformers empirically do well on a lot of machine learning problems, particularly where data has a sequential nature.