4 ms·
Google built transformers talk all you want about it's an ad company and da did da, in the end the whole LLM revolution is built on their tech
by VeejayRampay 3y ago
Google built transformers
talk all you want about it's an ad company and da did da, in the end the whole LLM revolution is built on their tech
- riku_iki 3y agoattention layer: core of transformer, was invented outside of Google.
- user_named 3y agoI thought that was pioneered in the paper that came out of Google?
- riku_iki 3y agoAttention was first proposed in this paper: https://arxiv.org/pdf/1409.0473.pdf https://arxiv.org/pdf/1409.0473.pdf and it is referenced from transformer paper.
- VeejayRampay 3y agoat this rate no one ever invents anything cause everything is built on math that some of the elements came out of somewhere else doesn't change the fact that Google came up with an impressive and game-changing piece of tech
- riku_iki 3y agoIt was not impressive at that time, because didn't achieve any quality breakthroughs. Later adapters of transformers like GPT and Bert achieved it with impressive results.
- rusticpenn 3y agoThe LLM revolution can be traced back to a 1996 paper from AT&T
- VeejayRampay 3y agoyou always build on top of something else and I'm not saying otherwise fact still remains that Google came up with an impressive and transformative piece of technology
- rusticpenn 3y agoThe groundbreaking paper from 96 did not lead to immediate advances because of the absence of computational power. This was then used by Hunton for the speech recognition paper which created the deep neural network revolution providing funding to create the transformers.