3 ms·
Most of the core idea of transformers was invented in the early 1990s, including what at the time were termed Fast Weight Programmers and are formally equivalen
by logicchains 3y ago
Most of the core idea of transformers was invented in the early 1990s, including what at the time were termed Fast Weight Programmers and are formally equivalent to linearised self-attention: https://people.idsia.ch/~juergen/fast-weight-programmer-1991-transformer.html https://people.idsia.ch/~juergen/fast-weight-programmer-1991... . Google just had the hardware to actually run and experiment with large transformers.
- tsunamifury 3y agoJesus, the arrogance here. Are HNers this ridiculously out of Touch. That is the investment and it was no small price. JUST is carrying a ton of weight here.
- camjw 3y agoAt least you concede then that Google invested in the hardware.
- 7373737373 3y agoIt would be very funny if at some point someone stood up to Schmidhuber and told him they already created something he did, but yet another 30 years earlier.