3 ms·
> But we're also definitely some way away from AIs doing their own research, and I'm not sure if scaling the current architecture alone will get us there. Hot
by throwup238 2y ago
> But we're also definitely some way away from AIs doing their own research, and I'm not sure if scaling the current architecture alone will get us there.
Hot take: I think it's fairly obvious the current architecture is not enough BUT it's already or almost at the point where it can be the backbone for a proper AGI/ASI system. Linking human communication and some light language-based reasoning to other AI models that handle the rest of the world model and reasoning, using special tokens to coordinate them.
I think it's just really early and most of this stuff hasn't even been attempted yet. There's tools/function calling which is kind of a proof of concept, and a lot of academic labs researching transformers applied to other fields like robotic control and learning, but no one's put it all together yet.
- lxgr 2y agoWhat's critically missing in the current architectures, in my view, is models that are able to "keep learning" (in a way beyond just providing a summary of the previous conversation, which grants surface-level insights at most). If we keep the current iteration cycle (retrain an LLM every couple of months, incorporating the current "status quo", including all of its previous conversations), we might get somewhat interesting results, but even the least motivated grad student has an iteration cycle orders of magnitude faster than that.