6 ms·

Once we start talking about multi-agent AI systems I think my intuitions go out the window. If we consider that GPT-4 is about as capable as a reasonably intell

Once we start talking about multi-agent AI systems I think my intuitions go out the window. If we consider that GPT-4 is about as capable as a reasonably intelligent high school graduate or college freshmen, I wonder how much actual benefit we get from multi-agent setups. I mean that intuitively I couldn't imagine replacing 1 experienced professional with 1, 2, 10, 100 or even 1000 intelligent high school graduates. Intelligence doesn't seem strictly additive across multiple individuals for all cases. But then I consider that one of the most powerful life-lines in the TV game show "Who wants to be a millionaire" was the "Ask the audience". I am reminded of the cliche of the wisdom of crowds, even when the crowd is made up of non-experts. This suggests to me that there is a kind of problem where multiple lower power agents can solve the issue to a higher quality. But there are also kinds of problem where a single higher-power intelligence will be necessary. I haven't developed an intuition when each approach is valid.

dmarchand90 3y ago
I suspect the real value is more from an architecture and caching perspective. Architecture as in you may have multiple GPT instances being prompted to look at a problem from different angles: "Analyse the problem as a pessimist", "As an optimist", "as a mathematician", "an engineer", "a philosopher", etc. Caching, as in determining what you can store from these outputs, e.g. "give a numeric 1-10 score of what you think of this product as a LGBT-friendly conservative from the Midwest", etc.
- zoogeny 3y ago
  > look at a problem from different angles Sure, but just as a thought experiment imagine you have a really bad disease. I give you two options: 100 high-school students can diagnose and prescribe a treatment or you can choose 1 professional with 10 years of experience in related diseases. You might initially prefer the professional with 10 years of experience. Would your opinion change if I told you that I selected the high-schoolers to be diverse so that one is pessimist, one is an optimist, one got good grades in engineering, one loves philosophy? Of course, my intuition might be wrong. For example, perhaps it legitimately would be better to have 100 high-school students where one is a high-school level ability with ear-nose-throat, one is a high-school level ability in oncology, one is a high-school level ability in cardiology, etc. Except some control AI would have to synthesize their answers into a coherent response ... and that controller would be high school level. I'm not really sold either way if I am honest. I don't think we have the answers to these questions. It just shows that my intuition about intelligence is open to challenges.
  - jxf 3y ago
    It's not quite the same thing, but the "mixture of experts" model is a popular deep learning approach that displays very good performance: see e.g., https://arxiv.org/abs/2208.02813 https://arxiv.org/abs/2208.02813
  - bagels 3y ago
    Do they have access to all of the information on the internet? Might give them a lot better chance.
  - dmarchand90 3y ago
    I suspect the divide may end up similar to that between a start up and a large company. Yes, some problems are better solved by a single person. But others benefit from a division of thought even if there is some bureaucratic overhead.
    - zoogeny 3y ago
      That is my speculation. But the question is one of intuition. If the speculation is correct, that some problems are better solved by a single more intelligent agent, then how can I determine the appropriate approach? What I mean to say is, if I am considering building a product based on LLMs then I may have to make a basic decision: can I use multiple cheap LLMs in a multi-agent setup or must I use a single expensive powerful LLM. Right now I don't have any intuition on what kinds of problems are solved most efficiently by either approach. Just looking at a problem description I can't intuit which approach is appropriate.
      - kian 3y ago
        This is a very interesting and important question. I think that until we answer your question rigorously, the best bet right now is to start with the expensive powerful LLM, and then after your system is prototyped and working, piecewise move prompts in your prompt network over to other systems as it proves feasible. If you try to start with the multiple cheap LLMs approach, it is almost certainly going to be more difficult to get the prompting right, and if you don't already know what you are trying to accomplish is feasible, you're adding a lot of work that might add up to nothing (even if it would actually work if you used the more powerful model).
beders 3y ago
GPT-4 is not comparable to a high school graduate. It will happily hallucinate facts and claim they are true. Well, ok, maybe it is ;) The issue they seem to tackle is trying to minimize hallucinations by injecting some human expertise into the pipeline. They do this by more strictly defining the roles and tasks a step in that pipeline needs to accomplish.
- two_in_one 3y ago
  > It will happily hallucinate facts and claim they are true. Well, ok, maybe it is ;) As if humans don't, bullshitting: https://www.sciencedirect.com/science/article/abs/pii/S0022103117305127 https://www.sciencedirect.com/science/article/abs/pii/S00221... https://www.poynter.org/fact-checking/2018/this-study-is-all-about-what-makes-people-bullshitters/ https://www.poynter.org/fact-checking/2018/this-study-is-all...
refulgentis 3y ago
Cosign, with extreme prejudice. We went through this with AutoGPT. I see two camps, people quietly working, and people working on grandiose ideas from the initial rush that make good headlines but not good products. Some informal warning signs I use subconciously: - "paper" on Arxiv, and its about prompt engineering - agent_S_ - state machine where the LLM is eating its own output repeatedly - LLM makes decisions based on its own output. There's 100x more alpha in the obvious stuff because even that's not well-implemented or shared widely. Ex. people are still stuck on hallucinations 90% of the time: an obvious way to handle that is doc retrieval. Last n.b.: in 2021 I was frustrated with ~100% of models being behind locked doors. Then, I saw how GPT3.0 was working and evolving. My mantra became "products not papers." Maybe that applies downstream of the LLM now.
- zoogeny 3y ago
  > My mantra became "products not papers." Maybe that applies downstream of the LLM now. I agree, in the sense that my intuition is fed by having as many available examples as possible. I am not even sure one could quantify the difference between intelligence levels in a way that I would be satisfied with in a paper. For example, I find it intuitive that a mixture of 10 narrowly fine-tuned GPT-3s are better at a task than 1 broadly fine-tuned GPT-3. But I don't have a real intuition about how many GPT-3s you would have to mix to match the quality of GPT-4, or if there even is any number of GPT-3s you can mix to achieve the result of GPT-4. I think we just need to start building systems and see what happens.
- tarun_anand 3y ago
  Are you saying AutoGPT has similar problems or has it solved it? We tried it recently and found it very challenging to get it to accomplish simple tasks.
  - refulgentis 3y ago
    TL;DR: AutoGPT has similar if not identical problems re: AutoGPT Got really excited at first. Thought maybe I had missed that it was viable in a year of playing with LLMs. Tried a few demos, didn't work. Looked into it more and confirmed a core loop involved LLM eating its own output to make decision. I still kept investigating it on my todo list, in case my earlier experiments with that approach were wrong. I took it off the todo list later. I saw near-universal feedback like yours, that general technique never worked IMHO, and IMHO sycophancy explains why. Paper here[1], TL;DR the model is very likely to agree, so critical feedback loops over multiple steps tend to settle into a loops of steps. IMHO this doesn't mean sycophancy breaks _all_ workflows, ex. a flow for writing a story involving outlining, writing, criticizing, then rewriting is a genuine real quality boost. However, if a human does write => criticize 10 times, it keeps getting better each iteration. If you have an LLM do it 10 times, IMHO it's actively harmful after round 3. "outline" => ["write page 1", "write page 2", "write page 3"] => ["feedback on page 1"..."feedback on page3" => ["use feedback and original draft to rewrite page1"..."page3"] => "combine pages 1 2 and 3 into cohesive story" [1] https://www.anthropic.com/index/discovering-language-model-behaviors-with-model-written-evaluations https://www.anthropic.com/index/discovering-language-model-b...
Closi 3y ago
> If we consider that GPT-4 is about as capable as a reasonably intelligent high school graduate or college freshmen, I wonder how much actual benefit we get from multi-agent setups. It's worth noting that GPT-4 internally uses a Mixture of Experts (MoE) model with 8 'experts' internally, so it's more similar to a multi-agent setup than you might think initially.
- capableweb 3y ago
  > It's worth noting that GPT-4 internally uses a Mixture of Experts (MoE) model with 8 'experts' internally Has this actually been confirmed, either officially by OpenAI or otherwise? As far as I know, George Hotz claimed this once, and since then everyone just assumed it was the truth without actually waiting for any sort of verification.
  - sheepscreek 3y ago
    TL;DR This might work - but it will be like watching Groundhog Day. It will require many iterations, make too many mistakes to get there and won’t remember a thing. My naive understanding of layers in a model is that each layer loosely acts as an expert in one step of the entire process. For example, in an object recognition model, one layer takes on the task of separating objects from the background, another excels at knowing the colour of different things, another might learn the difference between a blue sky vs. the colour sky blue. So essentially, we’re trying to mimic the same working model at a higher level of abstraction. Similar to how our body is made of atoms. Many atoms make a molecule. Many molecules make organic tissue, and amino acids that perform more complex operations. Skip all the way up and you have a human being. Human beings in numbers can put a man on the moon, create anti-matter and nuclear explosions. In theory - this can work just as well. One agent to provide a solution, another to critique it, another to verify it, another to mimic the end-user. The big obvious missing piece is memory and the ability to learn while doing (at least in existing LLMs). It’s like having a software development team that’s frozen in time and knowledge. These LLM agents will always require some micromanagement and hand holding, and will waste a lot of resources with failed attempts - that they are going to repeat every time you ask them to perform this task.
- 3y ago
hm-nah 3y ago
Mmmm… Assuming you’re referring to US high school, I’m going to say that GPT-4 is more useful than a high school graduate. Rationale: - Speed - Ability to recall - Depth and breadth of “exposure” I can’t speak to others high school experience, but in my experience, high school does not prepare young adults to be useful in the same settings that GPT-4 could be useful. Example prompt: - Pose this to a high school graduate and GPT-4, somehow benchmark quality. 1. “You are an experienced business strategist. Your client has the following problem <insert problem synopsis here>. Generate an outline of brainstorming topics your client should consider to accomplish <insert goal here>. - First, GPT-4 is going to respond in ~30sec or less - Second, a high school grad has 0 context for these types of prompts.
- ke88y 3y ago
  > Mmmm… Assuming you’re referring to US high school, I’m going to say that GPT-4 is more useful than a high school graduate. Useful for what? LLMs are largely useless for the vast majority of labor that people with HS degrees tend to do (which mostly involves not being behind a keyboard all day).
wefarrell 3y ago
> I mean that intuitively I couldn't imagine replacing 1 experienced professional with 1, 2, 10, 100 or even 1000 intelligent high school graduates This analogy doesn't make sense, because the professional is presumably also a high school graduate. This case is more like leveraging a team of specialists with expertise in different domains.
danielvaughn 3y ago
Interesting, multi-agent was almost my first thought when I began using ChatGPT. For complex tasks I figured you would want to chain together several LLMs, each for a dedicated domain. It would actually be really neat to be able to use a Zapier-like interface to build your own AI workflow.
djangelic 3y ago
Forgive my conjecture, as I'm no expert in this field, but have enjoyed playing with the different models. I see it as a matter of attention. GPT-4 is limited in the number of tokens you can feed it and receive back from it. I see this as how much attention the LLM can give you. I see meta-gpt and other models like this as increasing the attention of the LLM by allowing it combine it's short attention span with multiple assessments of the request, and give you a more complete picture of what you want because it can keep giving it consistent attention to the problem at hand, instead of simply trying to solve it on the first try. Someone please correct me if I'm way off, but this simple mental model helps me abstract this.
- hooande 3y ago
  The problem with this is that it has no memory across the different contexts. An analogy would be giving one page of a five page document to five different people, then taking it away and asking them to collaborate. While they can each give more attention to their individual page, none of them can see the whole picture and a lot of information will be lost when trying to communicate. You can use multiple agents, or split a lot of information across multiple requests to one agent. The result is the same. Some problems require a full understanding of the whole picture.
  - djangelic 3y ago
    That’s a great analogy thank you!
hooande 3y ago
The difference is that an LLM isn't 1000 different intelligences. It's one intelligence, being asked to pretend to be 1,000 different people. Every instance is the essentially the same weights trained on essentially the same data. The difference in perspective doesn't resemble that of the difference between any two humans. Humans love to think of multi agent systems as being like a team of people. It's much more like a writer imagining different characters and how they would respond. When George RR Martin imagines all 500 characters in Game of Thrones, there is a lot of diversity of perspective and thought there. But all of that is coming from one intelligence and doesn't represent a collaboration in any traditional sense.
- malaya_zemlya 3y ago
  I imagine you could still build a dream team of GPT3.5, GPT4, LLama2, Vicuna and their friends
  - RancheroBeans 3y ago
    This is something I’m working on, for now in a much more simplified format and still very rough. But I did manage to get team members to be contextually aware, so they know they team and everyone’s details. They can even search through chat threads with any given team member. https://www.conjure.team/ https://www.conjure.team/
- dragonwriter 3y ago
  > The difference is that an LLM isn't 1000 different intelligences. It's one intelligence, being asked to pretend to be 1,000 different people. No, its not. Its more like 1,000 individuals sharing one genetic template. Its just that the experiential context that along with the template makes an individual is very small with LLM instances.
- famouswaffles 3y ago
  >The difference is that an LLM isn't 1000 different intelligences. It's one intelligence, being asked to pretend to be 1,000 different people. No it's not. It does no good for a Language Model to configure a global persona. It needs to be able to predict text from wildly varying backgrounds and contexts. It's not pretending anymore than anything else it does is pretending. That's why experiments like the below actually work Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? (https://arxiv.org/abs/2301.07543 https://arxiv.org/abs/2301.07543) Out of One, Many: Using Language Models to Simulate Human Samples (https://arxiv.org/abs/2209.06899 https://arxiv.org/abs/2209.06899) A perfect LLM would predict Einstein as well as it would predict the dumbass down the street. Now RLHF does incentivize a more global persona by default but stepping away from that is trivial
_pdp_ 3y ago
Understanding a single node of the system is not the same as understanding the sum of all parts. In other words, a system of 2 high school graduates may appear at first as the sum of the output of just 2 high school graduates but the sum of 1000 high school graduates may appear as super intelligence. The phenomenon I am describing is known as "Emergence."
computerex 3y ago
I think it has to do with the fact that LLM's are stochastic, autoregressive models. There are "trajectories" that it can take that are wrong (hallucinations) and there are trajectories that lead to the right answer. In essence multi-agent is doing something like self consistency, self-reflection.
jhaenchen 3y ago
Different components of thought should be specialized. Choosing the meta structure will likely have an impact on what the system can achieve simply by the benefit of compartmentalization.
naveen99 3y ago
It’s the power to persuade (others to follow you). How do you decide which of the 100 agents is the expert ? They need to persuade the others. especially when they don’t know if they are the expert themselves. sometimes the right decision is to follow someone else.

dmarchand90 3y ago

I suspect the real value is more from an architecture and caching perspective. Architecture as in you may have multiple GPT instances being prompted to look at a problem from different angles: "Analyse the problem as a pessimist", "As an optimist", "as a mathematician", "an engineer", "a philosopher", etc. Caching, as in determining what you can store from these outputs, e.g. "give a numeric 1-10 score of what you think of this product as a LGBT-friendly conservative from the Midwest", etc.

zoogeny 3y ago
> look at a problem from different angles Sure, but just as a thought experiment imagine you have a really bad disease. I give you two options: 100 high-school students can diagnose and prescribe a treatment or you can choose 1 professional with 10 years of experience in related diseases. You might initially prefer the professional with 10 years of experience. Would your opinion change if I told you that I selected the high-schoolers to be diverse so that one is pessimist, one is an optimist, one got good grades in engineering, one loves philosophy? Of course, my intuition might be wrong. For example, perhaps it legitimately would be better to have 100 high-school students where one is a high-school level ability with ear-nose-throat, one is a high-school level ability in oncology, one is a high-school level ability in cardiology, etc. Except some control AI would have to synthesize their answers into a coherent response ... and that controller would be high school level. I'm not really sold either way if I am honest. I don't think we have the answers to these questions. It just shows that my intuition about intelligence is open to challenges.
- jxf 3y ago
  It's not quite the same thing, but the "mixture of experts" model is a popular deep learning approach that displays very good performance: see e.g., https://arxiv.org/abs/2208.02813 https://arxiv.org/abs/2208.02813
- bagels 3y ago
  Do they have access to all of the information on the internet? Might give them a lot better chance.
- dmarchand90 3y ago
  I suspect the divide may end up similar to that between a start up and a large company. Yes, some problems are better solved by a single person. But others benefit from a division of thought even if there is some bureaucratic overhead.
  - zoogeny 3y ago
    That is my speculation. But the question is one of intuition. If the speculation is correct, that some problems are better solved by a single more intelligent agent, then how can I determine the appropriate approach? What I mean to say is, if I am considering building a product based on LLMs then I may have to make a basic decision: can I use multiple cheap LLMs in a multi-agent setup or must I use a single expensive powerful LLM. Right now I don't have any intuition on what kinds of problems are solved most efficiently by either approach. Just looking at a problem description I can't intuit which approach is appropriate.
    - kian 3y ago
      This is a very interesting and important question. I think that until we answer your question rigorously, the best bet right now is to start with the expensive powerful LLM, and then after your system is prototyped and working, piecewise move prompts in your prompt network over to other systems as it proves feasible. If you try to start with the multiple cheap LLMs approach, it is almost certainly going to be more difficult to get the prompting right, and if you don't already know what you are trying to accomplish is feasible, you're adding a lot of work that might add up to nothing (even if it would actually work if you used the more powerful model).

beders 3y ago

GPT-4 is not comparable to a high school graduate. It will happily hallucinate facts and claim they are true. Well, ok, maybe it is ;) The issue they seem to tackle is trying to minimize hallucinations by injecting some human expertise into the pipeline. They do this by more strictly defining the roles and tasks a step in that pipeline needs to accomplish.

two_in_one 3y ago
> It will happily hallucinate facts and claim they are true. Well, ok, maybe it is ;) As if humans don't, bullshitting: https://www.sciencedirect.com/science/article/abs/pii/S0022103117305127 https://www.sciencedirect.com/science/article/abs/pii/S00221... https://www.poynter.org/fact-checking/2018/this-study-is-all-about-what-makes-people-bullshitters/ https://www.poynter.org/fact-checking/2018/this-study-is-all...

refulgentis 3y ago

Cosign, with extreme prejudice. We went through this with AutoGPT. I see two camps, people quietly working, and people working on grandiose ideas from the initial rush that make good headlines but not good products. Some informal warning signs I use subconciously: - "paper" on Arxiv, and its about prompt engineering - agent_S_ - state machine where the LLM is eating its own output repeatedly - LLM makes decisions based on its own output. There's 100x more alpha in the obvious stuff because even that's not well-implemented or shared widely. Ex. people are still stuck on hallucinations 90% of the time: an obvious way to handle that is doc retrieval. Last n.b.: in 2021 I was frustrated with ~100% of models being behind locked doors. Then, I saw how GPT3.0 was working and evolving. My mantra became "products not papers." Maybe that applies downstream of the LLM now.

zoogeny 3y ago
> My mantra became "products not papers." Maybe that applies downstream of the LLM now. I agree, in the sense that my intuition is fed by having as many available examples as possible. I am not even sure one could quantify the difference between intelligence levels in a way that I would be satisfied with in a paper. For example, I find it intuitive that a mixture of 10 narrowly fine-tuned GPT-3s are better at a task than 1 broadly fine-tuned GPT-3. But I don't have a real intuition about how many GPT-3s you would have to mix to match the quality of GPT-4, or if there even is any number of GPT-3s you can mix to achieve the result of GPT-4. I think we just need to start building systems and see what happens.
tarun_anand 3y ago
Are you saying AutoGPT has similar problems or has it solved it? We tried it recently and found it very challenging to get it to accomplish simple tasks.
- refulgentis 3y ago
  TL;DR: AutoGPT has similar if not identical problems re: AutoGPT Got really excited at first. Thought maybe I had missed that it was viable in a year of playing with LLMs. Tried a few demos, didn't work. Looked into it more and confirmed a core loop involved LLM eating its own output to make decision. I still kept investigating it on my todo list, in case my earlier experiments with that approach were wrong. I took it off the todo list later. I saw near-universal feedback like yours, that general technique never worked IMHO, and IMHO sycophancy explains why. Paper here[1], TL;DR the model is very likely to agree, so critical feedback loops over multiple steps tend to settle into a loops of steps. IMHO this doesn't mean sycophancy breaks _all_ workflows, ex. a flow for writing a story involving outlining, writing, criticizing, then rewriting is a genuine real quality boost. However, if a human does write => criticize 10 times, it keeps getting better each iteration. If you have an LLM do it 10 times, IMHO it's actively harmful after round 3. "outline" => ["write page 1", "write page 2", "write page 3"] => ["feedback on page 1"..."feedback on page3" => ["use feedback and original draft to rewrite page1"..."page3"] => "combine pages 1 2 and 3 into cohesive story" [1] https://www.anthropic.com/index/discovering-language-model-behaviors-with-model-written-evaluations https://www.anthropic.com/index/discovering-language-model-b...

Closi 3y ago

> If we consider that GPT-4 is about as capable as a reasonably intelligent high school graduate or college freshmen, I wonder how much actual benefit we get from multi-agent setups. It's worth noting that GPT-4 internally uses a Mixture of Experts (MoE) model with 8 'experts' internally, so it's more similar to a multi-agent setup than you might think initially.

capableweb 3y ago
> It's worth noting that GPT-4 internally uses a Mixture of Experts (MoE) model with 8 'experts' internally Has this actually been confirmed, either officially by OpenAI or otherwise? As far as I know, George Hotz claimed this once, and since then everyone just assumed it was the truth without actually waiting for any sort of verification.
- sheepscreek 3y ago
  TL;DR This might work - but it will be like watching Groundhog Day. It will require many iterations, make too many mistakes to get there and won’t remember a thing. My naive understanding of layers in a model is that each layer loosely acts as an expert in one step of the entire process. For example, in an object recognition model, one layer takes on the task of separating objects from the background, another excels at knowing the colour of different things, another might learn the difference between a blue sky vs. the colour sky blue. So essentially, we’re trying to mimic the same working model at a higher level of abstraction. Similar to how our body is made of atoms. Many atoms make a molecule. Many molecules make organic tissue, and amino acids that perform more complex operations. Skip all the way up and you have a human being. Human beings in numbers can put a man on the moon, create anti-matter and nuclear explosions. In theory - this can work just as well. One agent to provide a solution, another to critique it, another to verify it, another to mimic the end-user. The big obvious missing piece is memory and the ability to learn while doing (at least in existing LLMs). It’s like having a software development team that’s frozen in time and knowledge. These LLM agents will always require some micromanagement and hand holding, and will waste a lot of resources with failed attempts - that they are going to repeat every time you ask them to perform this task.
3y ago

hm-nah 3y ago

Mmmm… Assuming you’re referring to US high school, I’m going to say that GPT-4 is more useful than a high school graduate. Rationale: - Speed - Ability to recall - Depth and breadth of “exposure” I can’t speak to others high school experience, but in my experience, high school does not prepare young adults to be useful in the same settings that GPT-4 could be useful. Example prompt: - Pose this to a high school graduate and GPT-4, somehow benchmark quality. 1. “You are an experienced business strategist. Your client has the following problem <insert problem synopsis here>. Generate an outline of brainstorming topics your client should consider to accomplish <insert goal here>. - First, GPT-4 is going to respond in ~30sec or less - Second, a high school grad has 0 context for these types of prompts.

ke88y 3y ago
> Mmmm… Assuming you’re referring to US high school, I’m going to say that GPT-4 is more useful than a high school graduate. Useful for what? LLMs are largely useless for the vast majority of labor that people with HS degrees tend to do (which mostly involves not being behind a keyboard all day).

wefarrell 3y ago

> I mean that intuitively I couldn't imagine replacing 1 experienced professional with 1, 2, 10, 100 or even 1000 intelligent high school graduates This analogy doesn't make sense, because the professional is presumably also a high school graduate. This case is more like leveraging a team of specialists with expertise in different domains.

danielvaughn 3y ago

Interesting, multi-agent was almost my first thought when I began using ChatGPT. For complex tasks I figured you would want to chain together several LLMs, each for a dedicated domain. It would actually be really neat to be able to use a Zapier-like interface to build your own AI workflow.

djangelic 3y ago

Forgive my conjecture, as I'm no expert in this field, but have enjoyed playing with the different models. I see it as a matter of attention. GPT-4 is limited in the number of tokens you can feed it and receive back from it. I see this as how much attention the LLM can give you. I see meta-gpt and other models like this as increasing the attention of the LLM by allowing it combine it's short attention span with multiple assessments of the request, and give you a more complete picture of what you want because it can keep giving it consistent attention to the problem at hand, instead of simply trying to solve it on the first try. Someone please correct me if I'm way off, but this simple mental model helps me abstract this.

hooande 3y ago
The problem with this is that it has no memory across the different contexts. An analogy would be giving one page of a five page document to five different people, then taking it away and asking them to collaborate. While they can each give more attention to their individual page, none of them can see the whole picture and a lot of information will be lost when trying to communicate. You can use multiple agents, or split a lot of information across multiple requests to one agent. The result is the same. Some problems require a full understanding of the whole picture.
- djangelic 3y ago
  That’s a great analogy thank you!

hooande 3y ago

The difference is that an LLM isn't 1000 different intelligences. It's one intelligence, being asked to pretend to be 1,000 different people. Every instance is the essentially the same weights trained on essentially the same data. The difference in perspective doesn't resemble that of the difference between any two humans. Humans love to think of multi agent systems as being like a team of people. It's much more like a writer imagining different characters and how they would respond. When George RR Martin imagines all 500 characters in Game of Thrones, there is a lot of diversity of perspective and thought there. But all of that is coming from one intelligence and doesn't represent a collaboration in any traditional sense.

malaya_zemlya 3y ago
I imagine you could still build a dream team of GPT3.5, GPT4, LLama2, Vicuna and their friends
- RancheroBeans 3y ago
  This is something I’m working on, for now in a much more simplified format and still very rough. But I did manage to get team members to be contextually aware, so they know they team and everyone’s details. They can even search through chat threads with any given team member. https://www.conjure.team/ https://www.conjure.team/
dragonwriter 3y ago
> The difference is that an LLM isn't 1000 different intelligences. It's one intelligence, being asked to pretend to be 1,000 different people. No, its not. Its more like 1,000 individuals sharing one genetic template. Its just that the experiential context that along with the template makes an individual is very small with LLM instances.
famouswaffles 3y ago
>The difference is that an LLM isn't 1000 different intelligences. It's one intelligence, being asked to pretend to be 1,000 different people. No it's not. It does no good for a Language Model to configure a global persona. It needs to be able to predict text from wildly varying backgrounds and contexts. It's not pretending anymore than anything else it does is pretending. That's why experiments like the below actually work Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? (https://arxiv.org/abs/2301.07543 https://arxiv.org/abs/2301.07543) Out of One, Many: Using Language Models to Simulate Human Samples (https://arxiv.org/abs/2209.06899 https://arxiv.org/abs/2209.06899) A perfect LLM would predict Einstein as well as it would predict the dumbass down the street. Now RLHF does incentivize a more global persona by default but stepping away from that is trivial

_pdp_ 3y ago

Understanding a single node of the system is not the same as understanding the sum of all parts. In other words, a system of 2 high school graduates may appear at first as the sum of the output of just 2 high school graduates but the sum of 1000 high school graduates may appear as super intelligence. The phenomenon I am describing is known as "Emergence."

computerex 3y ago

I think it has to do with the fact that LLM's are stochastic, autoregressive models. There are "trajectories" that it can take that are wrong (hallucinations) and there are trajectories that lead to the right answer. In essence multi-agent is doing something like self consistency, self-reflection.

jhaenchen 3y ago

Different components of thought should be specialized. Choosing the meta structure will likely have an impact on what the system can achieve simply by the benefit of compartmentalization.

naveen99 3y ago

It’s the power to persuade (others to follow you). How do you decide which of the 100 agents is the expert ? They need to persuade the others. especially when they don’t know if they are the expert themselves. sometimes the right decision is to follow someone else.