3 ms·

People are trying to make these monolithic god models right now because everyone's chasing OpenAI's example with ChatGPT, but that approach is going to run out

People are trying to make these monolithic god models right now because everyone's chasing OpenAI's example with ChatGPT, but that approach is going to run out of steam for a lot of reasons. We're eventually going to end up with an agent model that does just what you say, recognizes specialized problem types such as math and calls the appropriate tool, such as a calculator or symbolic computation engine.

zozbot234 2y ago
Many models support function calling/RAG already, these are very similar features from a structural POV. But of course it's harder to train a model for such tasks, compared to just fitting some existing training set.
- CuriouslyC 2y ago
  I'm not sure having the LLM as the top level piece is the right approach though. Async is the direction we want to go, and LLMS are inherently synchronous. Additionally, LLMs are cumbersome to train and tune, having an agent that calls smaller models would unlock the power of the community to build customized models for specific needs.
  - szundi 2y ago
    We would like to have something like us. My daughter is just 3.5yrs old, she is inherently hard to train too, decades left. However, we find ourselves quite good.
    - CuriouslyC 2y ago
      You're assuming you're a monolith, but in fact you have many subnetworks in your brain. The neocortex acts to process information from other parts of the brain, and it itself comprised of a network of interacting modules.
szundi 2y ago
ChatGPT routinely does this already
- CuriouslyC 2y ago
  Yes, in a slow, synchronous, limited way.
meroes 2y ago
Hmm. And agent model for prompts we will ask K-12 math questions to? What are ultimately math questions often take lots of context to trudge through. It takes a solid core of natural language as well. I think there is always the need for a god model, because we want to speak in natural language as that’s easiest for us. I’m not a theoretician nor scientist, but train models on math and logical reasoning. Some very simple word problems contains tons of context, like family trees, implicit information, etc. and the process of step by step reasoning, not just the final answer, requires even more natural language processing.
- CuriouslyC 2y ago
  You don't need a god model to do that. You need a model that understands natural language really well and is good at identifying/extracting subtasks within a prompt, and then it needs to be able to dispatch those subproblems to the right place.

zozbot234 2y ago

Many models support function calling/RAG already, these are very similar features from a structural POV. But of course it's harder to train a model for such tasks, compared to just fitting some existing training set.

CuriouslyC 2y ago
I'm not sure having the LLM as the top level piece is the right approach though. Async is the direction we want to go, and LLMS are inherently synchronous. Additionally, LLMs are cumbersome to train and tune, having an agent that calls smaller models would unlock the power of the community to build customized models for specific needs.
- szundi 2y ago
  We would like to have something like us. My daughter is just 3.5yrs old, she is inherently hard to train too, decades left. However, we find ourselves quite good.
  - CuriouslyC 2y ago
    You're assuming you're a monolith, but in fact you have many subnetworks in your brain. The neocortex acts to process information from other parts of the brain, and it itself comprised of a network of interacting modules.

szundi 2y ago

ChatGPT routinely does this already

CuriouslyC 2y ago
Yes, in a slow, synchronous, limited way.

meroes 2y ago

Hmm. And agent model for prompts we will ask K-12 math questions to? What are ultimately math questions often take lots of context to trudge through. It takes a solid core of natural language as well. I think there is always the need for a god model, because we want to speak in natural language as that’s easiest for us. I’m not a theoretician nor scientist, but train models on math and logical reasoning. Some very simple word problems contains tons of context, like family trees, implicit information, etc. and the process of step by step reasoning, not just the final answer, requires even more natural language processing.

CuriouslyC 2y ago
You don't need a god model to do that. You need a model that understands natural language really well and is good at identifying/extracting subtasks within a prompt, and then it needs to be able to dispatch those subproblems to the right place.