3 ms·
This seems related to an interesting recent ACM ByteCast podcast episode with Edward Chang, an Adjunct Professor in the Department of Computer Science at Stanfo
by infogulch 3y ago
This seems related to an interesting recent ACM ByteCast podcast episode with Edward Chang, an Adjunct Professor in the Department of Computer Science at Stanford University. [1] (Note there is a transcript if you don't want to listen.)
The approach he uses is to arrange for multiple LLMs to dialogue between each other about a discussion topic where the human acts as a moderator instead of the question/answer format that LLMs commonly take today. They find that the final answer that multiple LLMs come to in dialogue results in a huge improvement in both precision and accuracy for the same resources.
[1]: https://learning.acm.org/bytecast/ep50-edward-y-chang https://learning.acm.org/bytecast/ep50-edward-y-chang
- j45 3y agoIs this effectively describing something like crewai?
- rahimnathwani 3y agoThis paper suggests that you don't need the debating part: just get LLM to work on the problem independently, and choose the most popular answer.
- infogulch 3y agoInteresting. Somehow it seems odd to add randomness (temperature) and then wash it away by averaging it out.
- ewild 3y agotemp applies to each token so the range of temperature is significantly larger than the average being pulled
- naasking 3y agoIn optimization problems, randomness can often get you out of local minima/maxima, and so averaging out a bunch of random search paths might get you better results in the worst case. Something similar might be happening here. The training set will be biased in various ways that might create weird local min/max points and so this process could avoid those weird kinks.
- xcv123 3y agoThe paper says that it enhances existing methods such as prompt engineering (chain of thought) and LLM debate. This agent method is orthogonal to LLM debate.
- swagmoney1606 3y agoI built something like this in Haskell! I never benchmarked it, but I actually found it quite compelling. I would define each agent as a different "expert" in a subdomain of mathematics for example: proof theorist, abstract algebraic expert, etc. I found it helpful, but the signal to noise ratio was high, lots of agents restating points etc.