3 ms·
> the research team I work with has about 1/20th the compute that a Google researcher has. This gives them a massive advantage The search space could easily be
by nyokodo 2y ago
> the research team I work with has about 1/20th the compute that a Google researcher has. This gives them a massive advantage
The search space could easily be so large that a less disciplined approach might yield fewer useful outcomes even with the compute advantage. Being forced to be focused and disciplined might actually be a big advantage to you.
- jahewson 2y agoIt’s hard to be disciplined about a black box though. That’s one reason why we’re all speeding off at a thousand miles per hour on transformers - the architecture works, why try other things?
- exe34 2y agowhoever comes up with the next trick could win big.
- jack_pp 2y agoCould, or meta makes you irrelevant with by reproducing your trick and giving it away for free
- kranke155 2y agoYann LeCun certainly seems to be one of the more interesting people in the space. He’s a notable skeptic of Llm intelligence and incredibly smart.
- exe34 2y agoTo be fair I don't think anybody saw the boom in LLMs coming from the initial Attention paper. At the time it was one of many ideas that sounded like they had potential.
- lallysingh 2y agoWhoever chooses the next winning lottery numbers could win big...
- exe34 2y agothat's kinda my point, we don't know what the next winning move will be.
- jumpCastle 2y agoAttention was invented because Bengio lab had to be disciplined about a black box (google had more compute)