3 ms·
I think it's unlikely that the first model to be widely considered AGI will be a transformer. Recent improvements to computational efficiency for attention mech
by tfehring 3y ago
I think it's unlikely that the first model to be widely considered AGI will be a transformer. Recent improvements to computational efficiency for attention mechanisms [0] seem to improve results a lot, as does RLHF, but neither is a paradigm shift like the introduction of transformers was. That's not to downplay their significance - that class of incremental improvements has driven a massive acceleration in AI capabilities in the last year - but I don't think it's ultimately how we'll get to AGI.
[0] https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn...
- sjdbaksbdhe 3y agoHow did we jump to AGI? "Sammy A thinks we've made the best engine with the tools at hand" -> "this will never get us out of the solar system" Sorry to unload on you. It is frustrating to constantly see AGI get brought up needlessly on HN
- tfehring 3y agoI'm using AGI here as arbitrary major improvement over the current state of the art. But given that OpenAI has the stated goal of creating AGI, I don't think it's a non-sequitur to respond to the parent comment's question > Are you saying instead, that concrete predictive algorithms need improvement or are we lumping the tuning into this? in the context of what's needed to get to AGI - just as if NASA built an engine we'd talk about its effectiveness in the context of space flight.