7 ms·

I'm not sure people in these comments are reading this paper correctly. This seems to essentially disprove the whole idea of multi-agent setups like Chain-of-t

by phire 3y ago

I'm not sure people in these comments are reading this paper correctly. This seems to essentially disprove the whole idea of multi-agent setups like Chain-of-thought and LLM-Debate. Because this paper introduces their alternative method which simply runs the same query multiple times on the same LLM, without any context shared across queries. And then they run a similarity algorithm on the answers and pick the most common answer. (Which makes sense to me. If an LLM is giving you a mixture of "hallucinations" and correct answers, the correct answers will similar and the hallucinations will hopefully be chaotic) And this simple algorithm preform just as well (and sometimes better) than all the other multi-agent algorithms. This suggests that the other multi-agent schemes with their clever prompts aren't really doing anything special; Their improve results are coming mostly from the fact that the LLM is run multiple times, that the prompt asks the LLM to pick the best answer.

sinuhe69 3y ago
But if I set the temperature to 0, the model will pick the highest probable token and the output will be always the same. But we already know that by no mean it can guarantee a correct answer. So how can multiple runs be better?
- phire 3y ago
  Yes, but picking the most similar output from a bunch of queries with a higher temperature is not the same thing as the output from a single low temperature query.
  - sinuhe69 3y ago
    Possibly, but it stills doesn’t explain why multiple runs will result in better answer. In the work, the authors also hasn’t compared the multiple runs results with the single run using zero temperature. So, maybe all the overhead is just to achieve the same result already encoded in the networks? I don’t know. Also the result is somewhat counterintuitive. We know that by low level of understanding, if we ask a student a hard question and he tried many times, the most accurate answer is often not the most popular one but a single answer. And that by retaining memory, reasoning capacity and continuous learning , which is not the case with LLM. Btw: HN is for discussion. If some just want to vote for the beauty contest, please leave.
    - phire 3y ago
      I found this other paper that tests Temperature: https://arxiv.org/abs/2402.05201 https://arxiv.org/abs/2402.05201 It appears that temperature has no impact on problem solving performance. So this paper isn't getting improved performance because the token for the correct answer is more probable. My theory is that the multiple queries are allowing the whole probability space of possible answers to be sampled. Not just the probabilities of the most likely output token, but the probabilities of all possible internal model states. And sampling that probability space of the whole model state and finding the average is a very different mathematical operation to just picking a single model state at random and then picking the most probable output tokens.
      - mlsu 3y ago
        I wonder if there is a clever/more efficient shortcut that could come from before the sample is taken on each token. We have the logits after all.
      - bt1a 3y ago
        If I'm reading this correctly, they had to discard Llama 2 answers and only use GPT-3.5 given answers to test the hypothesis. GPT-3.5 answering questions through the OAI API alone is not an acceptable method of testing problem solving ability across a range of temperatures. OpenAI does some blackbox wizardry on their end. There are many complex and clever sampling techniques for which temperature is just one (possibly dynamic) component One example from the llama.cpp codebase is dynamic temperature sampling https://github.com/ggerganov/llama.cpp/pull/4972/files https://github.com/ggerganov/llama.cpp/pull/4972/files Not sure what you mean by whole model state given that there are tens of thousands of possible tokens and the models have billions of parameters in XX,XXX-dimensional space. How many queries across how many sampling methods might you need? Err..how much time? :)
    - numpad0 3y ago
      Just from reading comments around, it feels intuitive to me that looking at a heatmap of cascading pendulum would be more “accurate” than looking at just one snapshot, and also that joints on the pendulums don’t necessarily need to be interlinked between iterations of simulations
- sp332 3y ago
  According to their code, they used temperature 1. https://anonymous.4open.science/r/more_agent_is_all_you_need/script/run_reasoning_task.sh https://anonymous.4open.science/r/more_agent_is_all_you_need...
zer00eyz 3y ago
>> Because this paper introduces their alternative method which simply runs the same query multiple times on the same LLM, without any context shared across queries. And then they run a similarity algorithm on the answers and pick the most common answer. https://en.wikipedia.org/wiki/Lorenz_system https://en.wikipedia.org/wiki/Lorenz_system Years ago weather simulations started tweaking input params and running their models over and over. Discarding outliers, taking averages. It works pretty well. Because LLM's mostly have random seeds (aka temperature) feeding them the same input and averaging the output is going to get you a better guess. Lorenz also gives some clues (if not an outright explanation) as to why the "hallucination" problem is likely unsolvable. If you buy into this line of thinking then it quickly becomes apparent that LLM's are more or less a dead end when it comes to AGI. Simulating isnt emulating... an LLM is as likely to become intelligent as a forecast is to control the weather.
- ekianjo 3y ago
  > LLM's are more or less a dead end when it comes to AGI. I don't think many people believe that LLMs are a way to AGI (whatever that actually means). But LLMs can still have many valid uses even if their prospects are limited in scope.
  - zer00eyz 3y ago
    >> I don't think many people believe that LLMs are a way to AGI Please tell Sam Altman ASAP Thanks!
    - dartos 3y ago
      You think he doesn’t know? Everything he says is marketing for OpenAI. Same as any other CEO with their company.
  - buu700 3y ago
    I recently read an interesting thread that laid out the case for LLMs being a path to AGI: https://old.reddit.com/r/singularity/comments/13ox85j/how_does_an_llm_even_work/jl6unp4 https://old.reddit.com/r/singularity/comments/13ox85j/how_do... The argument boils down to the idea that language isn't simply strings of words or bits of factual information, but an actual encoding of logic. By training statistical models on vast amounts of logic, we've given them a generalizable ability to perform logic. A sufficiently advanced LLM could thus potentially fulfill some definition of AGI. To be clear, this doesn't in any way imply that LLMs could ever fit the definition of artificial consciousness, which would be a completely different form of strong AI. They're effectively just mathematical functions (albeit extremely complicated ones), which simply take inputs and return outputs without any intervening subjective experience. Even if they can perform a complicated task, retrieve and effectively summarize complicated information, or say all the right things as a conversational partner, they have no concept of the meaning of their output. Maybe that limitation in itself puts a ceiling on their potential. Maybe the best possible LLM can only ever be 99.99% effective, and that 0.01% of the time it will go completely off the rails and disregard its instructions or hallucinate something ridiculous. Maybe the only way to overcome that is by keeping a human or a true artificial consciousness in the loop, in which case LLMs would still be extremely useful, but a flawed AGI if "AGI" at all. Or maybe a sufficiently advanced LLM and/or a sufficiently advanced error correction architecture will actually be enough to mitigate those issues. I don't have a strong opinion on where LLMs are ultimately headed, but I'm looking forward to seeing how it all unfolds. It's amazing how capabilities that were strictly in the realm of sci-fi so quickly became mundane.
whimsicalism 3y ago
how is chain-of-thought multi-agent?
- deleted 3y ago
  [deleted]
- Sharlin 3y ago
  How is what’s described here chain-of-thought?
  - xcv123 3y ago
    They were replying to this: > This seems to essentially disprove the whole idea of multi-agent setups like Chain-of-thought and LLM-Debate.
smusamashah 3y ago
Could multiple agents be used such that tokens emitted from LLM A is passed to B and output of B is passed to A meaning 2 agents will be being used to generate an output in a simple round Robin way? Both will share context in this case. My computer isn't big enough run two large models but this can be tried on tiny models perhaps. I realize that for more than two and very specialised agents this will require some intelligent way to pass the output to specialist agents only. And also this means that their must be some overlap between the agents.
- Sharlin 3y ago
  That is what’s already been done under the term "multi-agent". This paper argues that there’s no need for any such message-passing or context sharing, you just literally run the same query several times on the same model, fully independently, and then pick a "typical" reply according to some similarity metric.
  - xcv123 3y ago
    The paper says that it enhances multi-agent methods. It is not a replacement for that. It's an enhancement for existing methods.
    - Sharlin 3y ago
      Running the same query several times on the same model and taking the consensus opinion is still a multi-agent method.
m_kos 3y ago
I had a very similar idea a few months ago. I wanted to use this approach to have the LLM provide the probability that the generated answer is correct. The probability would simply be what fraction of all generated answers was the one selected. (Each generated answer would be generated with a different seed and the question would be of single choice kind.) The two issues I found were 1) the cost, 2) on some problems, LLMs can be wrong more often than they are not. Hopefully, as inference gets cheaper and of higher quality, someone will come up with a more feasible solution.
xcv123 3y ago
> I'm not sure people in these comments are reading this paper correctly. > This seems to essentially disprove the whole idea of multi-agent setups like Chain-of-thought and LLM-Debate. I'm not sure you have read the paper at all. Chain of thought prompting is not a multi-agent algorithm. The paper says that it enhances existing methods such as prompt engineering (chain of thought) and multi-agent debate. The sampling method presented in the paper is orthogonal to those methods.
0x008 3y ago
This is a very similar idea to ensemble models, which have been used for a long time in ML and proven to be very good. You average out the results of several predictors (or you let them vote and pick the most common prediction value), thereby reducing the noise in the prediction by choosing the common denominator of multiple predictions.
- sroussey 3y ago
  This is done in aerospace as well… however, even different teams clean room writing to the same spec have the tendency to make the same errors in their code, which ends up breaking the statistical model when this model was selected.
mirekrusin 3y ago
Good news is that you can use this setup for self supervised RL (artificial dreaming? increasing contrast?).
ben_w 3y ago
> Which makes sense to me. If an LLM is giving you a mixture of "hallucinations" and correct answers, the correct answers will similar and the hallucinations will hopefully be chaotic I expect that to give you something close to the confidence of the underlying model to some specific claim, which is good, but I still expect legends (urban and cultural) to be high-ranked. They'd be very human mistakes, but still mistakes. I think the only way past that is to build a world model, look for contradictions, and then look for new evidence to resolve those contradictions.
- dmarchand90 3y ago
  Be interesting to plug this into a bayesian optimization like framework: find out regions of language space where the models maximally disagree and then target those areas for extra training
Paradigma11 3y ago
My impression from github copilot is that hallucinations are the result of certain true facts having a low likelihood and copilot giving you the most likely answer anyway. Typically I have a certain library that does things in a very unorthodox and undocumented way and when I ask copilot for an example it gives me wonderful, totally understandable code of made up functions that I wouldnt need in the first place if the library worked that way. I dont think that running that query multiple times would help.
rasz 3y ago
>Which makes sense to me. If an LLM is giving you a mixture of "hallucinations" and correct answers, the correct answers will similar and the hallucinations will hopefully be chaotic Not my experience. I had multiple LLMs hallucinate hard when asked same question multiple times. The only way to break the cycle is to follow everything with questions demanding clarifications. "are you sure?" "this is wrong, correct the answer".
deleted 3y ago
[deleted]
auraai 3y ago
I don't think this type of method can scale indefinitely, it's essentially just "better" sampling within dense areas of knowledge space. It cannot help with better exploration outside these dense areas, because these explorations won't have a consensus among agents almost by definition.

by phire 3y ago

sinuhe69 3y ago

But if I set the temperature to 0, the model will pick the highest probable token and the output will be always the same. But we already know that by no mean it can guarantee a correct answer. So how can multiple runs be better?

phire 3y ago
Yes, but picking the most similar output from a bunch of queries with a higher temperature is not the same thing as the output from a single low temperature query.
- sinuhe69 3y ago
  Possibly, but it stills doesn’t explain why multiple runs will result in better answer. In the work, the authors also hasn’t compared the multiple runs results with the single run using zero temperature. So, maybe all the overhead is just to achieve the same result already encoded in the networks? I don’t know. Also the result is somewhat counterintuitive. We know that by low level of understanding, if we ask a student a hard question and he tried many times, the most accurate answer is often not the most popular one but a single answer. And that by retaining memory, reasoning capacity and continuous learning , which is not the case with LLM. Btw: HN is for discussion. If some just want to vote for the beauty contest, please leave.
  - phire 3y ago
    I found this other paper that tests Temperature: https://arxiv.org/abs/2402.05201 https://arxiv.org/abs/2402.05201 It appears that temperature has no impact on problem solving performance. So this paper isn't getting improved performance because the token for the correct answer is more probable. My theory is that the multiple queries are allowing the whole probability space of possible answers to be sampled. Not just the probabilities of the most likely output token, but the probabilities of all possible internal model states. And sampling that probability space of the whole model state and finding the average is a very different mathematical operation to just picking a single model state at random and then picking the most probable output tokens.
    - mlsu 3y ago
      I wonder if there is a clever/more efficient shortcut that could come from before the sample is taken on each token. We have the logits after all.
    - bt1a 3y ago
      If I'm reading this correctly, they had to discard Llama 2 answers and only use GPT-3.5 given answers to test the hypothesis. GPT-3.5 answering questions through the OAI API alone is not an acceptable method of testing problem solving ability across a range of temperatures. OpenAI does some blackbox wizardry on their end. There are many complex and clever sampling techniques for which temperature is just one (possibly dynamic) component One example from the llama.cpp codebase is dynamic temperature sampling https://github.com/ggerganov/llama.cpp/pull/4972/files https://github.com/ggerganov/llama.cpp/pull/4972/files Not sure what you mean by whole model state given that there are tens of thousands of possible tokens and the models have billions of parameters in XX,XXX-dimensional space. How many queries across how many sampling methods might you need? Err..how much time? :)
  - numpad0 3y ago
    Just from reading comments around, it feels intuitive to me that looking at a heatmap of cascading pendulum would be more “accurate” than looking at just one snapshot, and also that joints on the pendulums don’t necessarily need to be interlinked between iterations of simulations
sp332 3y ago
According to their code, they used temperature 1. https://anonymous.4open.science/r/more_agent_is_all_you_need/script/run_reasoning_task.sh https://anonymous.4open.science/r/more_agent_is_all_you_need...

zer00eyz 3y ago

>> Because this paper introduces their alternative method which simply runs the same query multiple times on the same LLM, without any context shared across queries. And then they run a similarity algorithm on the answers and pick the most common answer. https://en.wikipedia.org/wiki/Lorenz_system https://en.wikipedia.org/wiki/Lorenz_system Years ago weather simulations started tweaking input params and running their models over and over. Discarding outliers, taking averages. It works pretty well. Because LLM's mostly have random seeds (aka temperature) feeding them the same input and averaging the output is going to get you a better guess. Lorenz also gives some clues (if not an outright explanation) as to why the "hallucination" problem is likely unsolvable. If you buy into this line of thinking then it quickly becomes apparent that LLM's are more or less a dead end when it comes to AGI. Simulating isnt emulating... an LLM is as likely to become intelligent as a forecast is to control the weather.

ekianjo 3y ago
> LLM's are more or less a dead end when it comes to AGI. I don't think many people believe that LLMs are a way to AGI (whatever that actually means). But LLMs can still have many valid uses even if their prospects are limited in scope.
- zer00eyz 3y ago
  >> I don't think many people believe that LLMs are a way to AGI Please tell Sam Altman ASAP Thanks!
  - dartos 3y ago
    You think he doesn’t know? Everything he says is marketing for OpenAI. Same as any other CEO with their company.
- buu700 3y ago
  I recently read an interesting thread that laid out the case for LLMs being a path to AGI: https://old.reddit.com/r/singularity/comments/13ox85j/how_does_an_llm_even_work/jl6unp4 https://old.reddit.com/r/singularity/comments/13ox85j/how_do... The argument boils down to the idea that language isn't simply strings of words or bits of factual information, but an actual encoding of logic. By training statistical models on vast amounts of logic, we've given them a generalizable ability to perform logic. A sufficiently advanced LLM could thus potentially fulfill some definition of AGI. To be clear, this doesn't in any way imply that LLMs could ever fit the definition of artificial consciousness, which would be a completely different form of strong AI. They're effectively just mathematical functions (albeit extremely complicated ones), which simply take inputs and return outputs without any intervening subjective experience. Even if they can perform a complicated task, retrieve and effectively summarize complicated information, or say all the right things as a conversational partner, they have no concept of the meaning of their output. Maybe that limitation in itself puts a ceiling on their potential. Maybe the best possible LLM can only ever be 99.99% effective, and that 0.01% of the time it will go completely off the rails and disregard its instructions or hallucinate something ridiculous. Maybe the only way to overcome that is by keeping a human or a true artificial consciousness in the loop, in which case LLMs would still be extremely useful, but a flawed AGI if "AGI" at all. Or maybe a sufficiently advanced LLM and/or a sufficiently advanced error correction architecture will actually be enough to mitigate those issues. I don't have a strong opinion on where LLMs are ultimately headed, but I'm looking forward to seeing how it all unfolds. It's amazing how capabilities that were strictly in the realm of sci-fi so quickly became mundane.

whimsicalism 3y ago

how is chain-of-thought multi-agent?

deleted 3y ago
[deleted]
Sharlin 3y ago
How is what’s described here chain-of-thought?
- xcv123 3y ago
  They were replying to this: > This seems to essentially disprove the whole idea of multi-agent setups like Chain-of-thought and LLM-Debate.

smusamashah 3y ago

Could multiple agents be used such that tokens emitted from LLM A is passed to B and output of B is passed to A meaning 2 agents will be being used to generate an output in a simple round Robin way? Both will share context in this case. My computer isn't big enough run two large models but this can be tried on tiny models perhaps. I realize that for more than two and very specialised agents this will require some intelligent way to pass the output to specialist agents only. And also this means that their must be some overlap between the agents.

Sharlin 3y ago
That is what’s already been done under the term "multi-agent". This paper argues that there’s no need for any such message-passing or context sharing, you just literally run the same query several times on the same model, fully independently, and then pick a "typical" reply according to some similarity metric.
- xcv123 3y ago
  The paper says that it enhances multi-agent methods. It is not a replacement for that. It's an enhancement for existing methods.
  - Sharlin 3y ago
    Running the same query several times on the same model and taking the consensus opinion is still a multi-agent method.

m_kos 3y ago

I had a very similar idea a few months ago. I wanted to use this approach to have the LLM provide the probability that the generated answer is correct. The probability would simply be what fraction of all generated answers was the one selected. (Each generated answer would be generated with a different seed and the question would be of single choice kind.) The two issues I found were 1) the cost, 2) on some problems, LLMs can be wrong more often than they are not. Hopefully, as inference gets cheaper and of higher quality, someone will come up with a more feasible solution.

xcv123 3y ago

> I'm not sure people in these comments are reading this paper correctly. > This seems to essentially disprove the whole idea of multi-agent setups like Chain-of-thought and LLM-Debate. I'm not sure you have read the paper at all. Chain of thought prompting is not a multi-agent algorithm. The paper says that it enhances existing methods such as prompt engineering (chain of thought) and multi-agent debate. The sampling method presented in the paper is orthogonal to those methods.

0x008 3y ago

This is a very similar idea to ensemble models, which have been used for a long time in ML and proven to be very good. You average out the results of several predictors (or you let them vote and pick the most common prediction value), thereby reducing the noise in the prediction by choosing the common denominator of multiple predictions.

sroussey 3y ago
This is done in aerospace as well… however, even different teams clean room writing to the same spec have the tendency to make the same errors in their code, which ends up breaking the statistical model when this model was selected.

mirekrusin 3y ago

Good news is that you can use this setup for self supervised RL (artificial dreaming? increasing contrast?).

ben_w 3y ago

> Which makes sense to me. If an LLM is giving you a mixture of "hallucinations" and correct answers, the correct answers will similar and the hallucinations will hopefully be chaotic I expect that to give you something close to the confidence of the underlying model to some specific claim, which is good, but I still expect legends (urban and cultural) to be high-ranked. They'd be very human mistakes, but still mistakes. I think the only way past that is to build a world model, look for contradictions, and then look for new evidence to resolve those contradictions.

dmarchand90 3y ago
Be interesting to plug this into a bayesian optimization like framework: find out regions of language space where the models maximally disagree and then target those areas for extra training

Paradigma11 3y ago

My impression from github copilot is that hallucinations are the result of certain true facts having a low likelihood and copilot giving you the most likely answer anyway. Typically I have a certain library that does things in a very unorthodox and undocumented way and when I ask copilot for an example it gives me wonderful, totally understandable code of made up functions that I wouldnt need in the first place if the library worked that way. I dont think that running that query multiple times would help.

rasz 3y ago

>Which makes sense to me. If an LLM is giving you a mixture of "hallucinations" and correct answers, the correct answers will similar and the hallucinations will hopefully be chaotic Not my experience. I had multiple LLMs hallucinate hard when asked same question multiple times. The only way to break the cycle is to follow everything with questions demanding clarifications. "are you sure?" "this is wrong, correct the answer".

deleted 3y ago

[deleted]

auraai 3y ago

I don't think this type of method can scale indefinitely, it's essentially just "better" sampling within dense areas of knowledge space. It cannot help with better exploration outside these dense areas, because these explorations won't have a consensus among agents almost by definition.