4 ms·
Investment is not the sole input into creating technology, time matters to. The tranfsormer paper probably didn't take much investment to create, but without it
by anonylizard 3y ago
Investment is not the sole input into creating technology, time matters to. The tranfsormer paper probably didn't take much investment to create, but without it, any attempts at creating LLMs would have been futile no matter the billions poured in.
Moreover, technology is created as spillover effect from investments in other areas (That aren't speculative). CUDA is only possible because Nvidia was supported by gaming revenue for a decade. Without CUDA, there is no alexnet, and no deep learning boom will happen if we are still on CPUs or google's proprietary TPUs.
- tsunamifury 3y agoI’m sorry … but what?! As someone who worked on early LLM tech at Google hundreds of millions were poured into it. Do you think that paper just spontaneously came from pure theory?
- logicchains 3y agoMost of the core idea of transformers was invented in the early 1990s, including what at the time were termed Fast Weight Programmers and are formally equivalent to linearised self-attention: https://people.idsia.ch/~juergen/fast-weight-programmer-1991-transformer.html https://people.idsia.ch/~juergen/fast-weight-programmer-1991... . Google just had the hardware to actually run and experiment with large transformers.
- tsunamifury 3y agoJesus, the arrogance here. Are HNers this ridiculously out of Touch. That is the investment and it was no small price. JUST is carrying a ton of weight here.
- camjw 3y agoAt least you concede then that Google invested in the hardware.
- 7373737373 3y agoIt would be very funny if at some point someone stood up to Schmidhuber and told him they already created something he did, but yet another 30 years earlier.