3 ms·

There has been a huge mistake getting made in the past few months almost everywhere I've seen AI being discussed, which is to conflate ChatGPT with all of AI.

• CuriouslyC 4y ago
  The way this is going to go is that people will build a lot of specialized models, and we'll have a central executive model which will probably evolve from a LLM, which maps query -> response, except the response space will include performing tasks and constructing (potentially multimedia) structured data. The central executive will dispatch to other models and knit the results together. These models are memoryless. In order to have memory, you either need to either be able to provide them as context to the query, or provide a mechanism for users to have custom weights applied to models. Providing context to a query is just a matter of increasing token limits and fine tuning the model on "query + context" data so it uses the context correctly. Allowing users to have custom weights for models is much more computationally expensive, to the point that I don't think this method can be reasonably scaled as a service. I think the only way custom weights per user can work is if the central executive calls fine tuned helper models on the client machine.
  • jerf 4y ago
    This is what I'm talking about. There are and will continue to be AIs that aren't language models at all. Do not incorporate them, do not use them, are not based on them, do not have the same characteristics. They already exist and already affect our lives and world. Large Language Models are not equal to AI. Making conclusions about "AI" from "LLM"s is not valid. And the day will come when some other model entirely outperforms the LLMs. Personally I think there's a bit of a selection process in play here, where they are the most flashy and impressive AI models to date, but if you get down to it, they are not in fact very useful. Almost every use I've seen proposed for the LLMs range from completely impractical to "yeah, it'll sort of work, but you'll never really be able to count on your LLM to do that, it confabulates too much and too freely and it's not something you can fix because confabulation is the very foundation of the technology". Honestly if I had to guess LLMs will be seen to be a dead end in another 10 or 20 years. AI will not. (Or, to put it another way, LLMs are the AI equivalent of a demo-scene demo. The demos look very impressive. But if you try to naively look at a great demoscene demo and then come to conclusions based on that demo as to what the underlying hardware is capable of, you will be hugely misled. What you see is not the normal capability of the system, you see a hyper-optimized system being driven to its absolute limits to achieve the visible results you see, and it can't do anything else because it has already "spent" all of the engineering budget available. I see LLMs very similarly. Very impressive. Nowhere near as useful as the demo may make you superficially think, though. If I were trying to build an "AI startup", I would be zigging to everybody else's zag and looking at what else the AI world is producing; I think there's far less potential in the LLMs than meets the eye.) Unfortunately the things the LLMs are best at will be flooding the internet with an infinite stream of directed content and manifesting the Dead Internet theory. Edit: Here's a concrete example. Consider Boston Dynamic's robots. Do they run on a Large Language Model? No. Of course not. The question doesn't even hardly make sense. But whatever they will be doing, they do with AI. ChatGPT can't kill you, except through a very lengthy and strained series of events. Boston Dynamic's robots and their fellows can.
    • CuriouslyC 4y ago
      My original point was that language models will almost certainly be the glue that holds a variety of models together in a post-AI world. We communicate with other humans using language, we will use language with AI as well. In order to understand and debug model networks they will probably be built using a common embedding framework that allows you to derive textual descriptions of the things being passed between them. LLMs already have the ability to generate text to dispatch to other services and synthesize results, all that is required for it to act as an "model executive" is a framework that can ensure security/correctness of code before building and executing it. Even if LLMs never progressed past the current crop of transformers, there are a lot of small architecture optimizations that could reap big performance improvements, and compute/data scaling alone basically guarantees a level of functionality sufficient for a lot of very interesting applications. I agree that trying to scale larger and larger language models to build "the one model to rule them all" is probably not going to play out.