3 ms·
> I also wonder - is the compute the main limiting factor today ? So much of this is a blackbox that you have to wing it for lots of things and try stuff. The
by leetharris 2y ago
> I also wonder - is the compute the main limiting factor today ?
So much of this is a blackbox that you have to wing it for lots of things and try stuff. The more compute you have, the more YOLO runs you can do.
For example, the research team I work with has about 1/20th the compute that a Google researcher has. This gives them a massive advantage because they can afford to train lots of random ideas and see what kind of advantage they get. We have to be much, much more measured and predict our outcomes better.
- nyokodo 2y ago> the research team I work with has about 1/20th the compute that a Google researcher has. This gives them a massive advantage The search space could easily be so large that a less disciplined approach might yield fewer useful outcomes even with the compute advantage. Being forced to be focused and disciplined might actually be a big advantage to you.
- jahewson 2y agoIt’s hard to be disciplined about a black box though. That’s one reason why we’re all speeding off at a thousand miles per hour on transformers - the architecture works, why try other things?
- exe34 2y agowhoever comes up with the next trick could win big.
- jack_pp 2y agoCould, or meta makes you irrelevant with by reproducing your trick and giving it away for free
- kranke155 2y agoYann LeCun certainly seems to be one of the more interesting people in the space. He’s a notable skeptic of Llm intelligence and incredibly smart.
- exe34 2y agoTo be fair I don't think anybody saw the boom in LLMs coming from the initial Attention paper. At the time it was one of many ideas that sounded like they had potential.
- lallysingh 2y agoWhoever chooses the next winning lottery numbers could win big...
- exe34 2y agothat's kinda my point, we don't know what the next winning move will be.
- jumpCastle 2y agoAttention was invented because Bengio lab had to be disciplined about a black box (google had more compute)
- TheAlchemist 2y agoWhat's the process of deciding what stuff you will try ?
- thecupisblue 2y ago> So much of this is a blackbox that you have to wing it for lots of things and try stuff Isn't OP talking about this _blackbox_ of gains? But this is highly disappointing too - there are speed gains and tradeoffs to be had at every blackbox layer, but instead of doing that and actually gaining these improvements across all future experiments, we do YOLO runs and don't garner them at all.