5 ms·
All the LC grinding may come in handy after all! /s What algorithms specifically show the most results upon improvement? Going into this I thought the jump of
by no_wizard 4y ago
All the LC grinding may come in handy after all! /s
What algorithms specifically show the most results upon improvement? Going into this I thought the jump of improvements were really related more advanced automated tuning and result correction, in which it could be done at scale as it were allowing a small team of data scientists to tweak the models until desired results were being achieved.
Are you saying instead, that concrete predictive algorithms need improvement or are we lumping the tuning into this?
- junipertea 4y agoWe need more data efficient neural network architectures. Transformers work exceptionally well because they allow us to just dump more data into it, but ultimately we want to learn advanced behavior without having to feed it Shakespeare
- uoaei 4y agoInductive Bias Is All You Need
- tfehring 4y agoI think it's unlikely that the first model to be widely considered AGI will be a transformer. Recent improvements to computational efficiency for attention mechanisms [0] seem to improve results a lot, as does RLHF, but neither is a paradigm shift like the introduction of transformers was. That's not to downplay their significance - that class of incremental improvements has driven a massive acceleration in AI capabilities in the last year - but I don't think it's ultimately how we'll get to AGI. [0] https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn...
- sjdbaksbdhe 4y agoHow did we jump to AGI? "Sammy A thinks we've made the best engine with the tools at hand" -> "this will never get us out of the solar system" Sorry to unload on you. It is frustrating to constantly see AGI get brought up needlessly on HN
- tfehring 4y agoI'm using AGI here as arbitrary major improvement over the current state of the art. But given that OpenAI has the stated goal of creating AGI, I don't think it's a non-sequitur to respond to the parent comment's question > Are you saying instead, that concrete predictive algorithms need improvement or are we lumping the tuning into this? in the context of what's needed to get to AGI - just as if NASA built an engine we'd talk about its effectiveness in the context of space flight.
- goldenManatee 4y agobubble sort /s
- uoaei 4y agoTraditional CS may have something to do with slightly improving the performance by allowing more training for the same compute, but it won't be an order of magnitude or more. The improvements to be gained will be found more in statistics than CS per se.
- jacobr1 4y agoI'm not sure. Methods like Chinchilla and Quantization have been able to reduce compute by more than an order of magnitude. There might very well be a few more levels of optimizations within the same statistical paradigm.