35 ms·
I'm interested how an innovation like this affects the business prospects. Let's assume this is a paradigm shift on the scale of Transformers / `Attention is a
by NotAnOtter 1y ago
I'm interested how an innovation like this affects the business prospects.
Let's assume this is a paradigm shift on the scale of Transformers / `Attention is all you need`. Companies build out new models and pump another $100 Billion through it. And then a year from now, another innovation comes out. Same circus. And again.
No one wants to be left behind but trying to keep up will sink smaller companies.
- curious_cat_163 1y agoI am not sure why this ought to require "pump another $100 Billion". Could you elaborate? Yes, the more recent generation of GPUs optimize for attention math. But they are still fairly "general-purpose" accelerators as well. So when I see papers like this (interesting idea, btw!), my mental model for costs suggests that the CapEx to buy up the GPUs and build out the data centers would get re-used for this and 100s of other ideas and experiments. And then the hope is that the best ideas will occupy more of the available capacity...
- gessha 1y agoSir, this is an arxiv paper
- NotAnOtter 1y agoSo true, just like this one: https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762