8 ms·
MetaGPT: Meta Programming for Multi-Agent Collaborative Framework
- monkeydust 3y agoI'm curious, is anyone using any kind of LLM agent workflow in a client facing production environment?
- TuringNYC 3y ago>> I'm curious, is anyone using any kind of LLM agent workflow in a client facing production environment? Yes. Not everyone will admit as most big companies have policies against it. Small companies are, because the cost saving is so high.
- TrickardRixx 3y agoMy employer is. The LLM agent did not replace any humans, though. It was created as a new value-add service for clients.
- drunkencoder 3y agoYes, recently upscaled research for my iOS app. Have made a pipeline for fact extracts from various sources, including some sentiment analysis. Would have taken me years to do it manually. Will mainly be value adding for end user . Not directly client facing since this is done offline and shipped as data into the final product.
- Lambdanaut 3y agoYes, in public safety to make response suggestions to emergency operators in time critical situations when texting is the mode of communication.
- ativzzz 3y agoThis seems... like a terrible idea on the surface. The cost of LLM hallucination in a time critical crisis situation seems immense
- jxf 3y agoYes. We've launched multiple production instances of this across a handful of clients. In general humans using the new workflows show very significant performance and quality improvements over unassisted humans so far for the well-defined tasks the workflows are tailored to perform.
- TheObviousOne 3y agocan you elaborate?
- jxf 3y agoProbably not without violating an NDA. What would you like to know?
- TheObviousOne 3y agoAWWWW... It would be great to share some "meta-data" on the usecase - i wonder whether it is narrow task (that can be done via CRON job), or something that are more general and can be creative and not in control sometimes. What kind of tools that the agent use? (Web-search/ Generating content/...) Did you made any modification to the code or use it "as is". if so- how much time did you spend on the modification and in what kind of areas... Feel free to answer only subset of the questions that you feel comfortable with them.
- marviel 3y agoYes, the key is to implement clear roles, responsibilities, relationships, and communication channels, as the OP article explains.
- tsunamifury 3y agoallofus.ai our entire focus is on multi-agent workflows based and trained on real peoples published work. I think the output is amazing, but I'm biased of course. Its ability to accomplish soft tasks like advice, help etc is far superior to vanilla GPT4
- thatcherthorn 3y agoTo me, the recent breakthroughs in generative AI seem as big as when the Internet or the smartphone became generally available.
- jsemrau 3y agoExpand please. I think the initial hype is gone and we see that most implementations are not that much of a game changer just yet. Stable Diffusion can create images but its still incredibly cumbersome to find the right seed and prompt to get the image one desires. ControlNet, Inpainting, and Loras are helpful but have to be implemented in a useful workflow.
- thatcherthorn 3y agoI agree that the hype has dissipated. I'd probably contribute a large part of that to media cycles though. Personally, I think the use cases that go beyond chat are going to be the most valuable. Specifically, the ability to produce structured information.
- sdenton4 3y agoWell, the internet had a hype-cycle as well... It went from very exciting to a niche for weirdoes to completely ubiquitous.
- deleted 3y ago[deleted]
- zoogeny 3y agoOnce we start talking about multi-agent AI systems I think my intuitions go out the window. If we consider that GPT-4 is about as capable as a reasonably intelligent high school graduate or college freshmen, I wonder how much actual benefit we get from multi-agent setups. I mean that intuitively I couldn't imagine replacing 1 experienced professional with 1, 2, 10, 100 or even 1000 intelligent high school graduates. Intelligence doesn't seem strictly additive across multiple individuals for all cases. But then I consider that one of the most powerful life-lines in the TV game show "Who wants to be a millionaire" was the "Ask the audience". I am reminded of the cliche of the wisdom of crowds, even when the crowd is made up of non-experts. This suggests to me that there is a kind of problem where multiple lower power agents can solve the issue to a higher quality. But there are also kinds of problem where a single higher-power intelligence will be necessary. I haven't developed an intuition when each approach is valid.
- dmarchand90 3y agoI suspect the real value is more from an architecture and caching perspective. Architecture as in you may have multiple GPT instances being prompted to look at a problem from different angles: "Analyse the problem as a pessimist", "As an optimist", "as a mathematician", "an engineer", "a philosopher", etc. Caching, as in determining what you can store from these outputs, e.g. "give a numeric 1-10 score of what you think of this product as a LGBT-friendly conservative from the Midwest", etc.
- zoogeny 3y ago> look at a problem from different angles Sure, but just as a thought experiment imagine you have a really bad disease. I give you two options: 100 high-school students can diagnose and prescribe a treatment or you can choose 1 professional with 10 years of experience in related diseases. You might initially prefer the professional with 10 years of experience. Would your opinion change if I told you that I selected the high-schoolers to be diverse so that one is pessimist, one is an optimist, one got good grades in engineering, one loves philosophy? Of course, my intuition might be wrong. For example, perhaps it legitimately would be better to have 100 high-school students where one is a high-school level ability with ear-nose-throat, one is a high-school level ability in oncology, one is a high-school level ability in cardiology, etc. Except some control AI would have to synthesize their answers into a coherent response ... and that controller would be high school level. I'm not really sold either way if I am honest. I don't think we have the answers to these questions. It just shows that my intuition about intelligence is open to challenges.
- famouswaffles 3y agoPapers that try to implement a multi agent system https://arxiv.org/abs/2307.07924 https://arxiv.org/abs/2307.07924 https://arxiv.org/abs/2307.02485 https://arxiv.org/abs/2307.02485
- joenot443 3y agoI'm gonna take some time this afternoon to give this a spin. The challenge is a multiplayer Snake game set on an expanding canvas which can support an arbitrary amount of players and communicates over websocket. Players join through the browser and control their snake with the keyboard. This was a take-home challenge I got back in 2015 for an internship. I spent all day on it and had a blast. Curious to compare my nooby college code to state of the art LLM code!
- sgslo 3y agoI ran the 'create a snake' game example using the paper's source at https://github.com/geekan/MetaGPT https://github.com/geekan/MetaGPT The creation process used 11,940 tokens on input and 2,993 tokens on output, which cost $0.35 and $0.18, respectively. The game it generated consisted of four python classes in four separate files: Main, Game, Snake, and Food. The game executed without error on the first try, but the snake wasn't able to 'eat' the food. Here's the relevant code for 'eating' food: # Check if the snake ate the food if self.snake.body[0] == self.food.position: self.score += 1 self.snake.grow() self.food.generate() The issue was that the snake's body was represented as a list of lists, whereas the food position was stored in a tuple. After changing the food position to a list, the game worked correctly.
- reedf1 3y agoWhich positions did you "hire"? Would be interested to see if that changes the robustness of the output. Same goes for "investment".
- 33a 3y agoFor comparison, I asked ChatGPT to build one and here's what I got in one shot: https://chat.openai.com/share/b4b399ef-1def-4f68-b2f1-8c56ca3858ae https://chat.openai.com/share/b4b399ef-1def-4f68-b2f1-8c56ca... Seems to work correctly, didn't have to change anything in the code. Like all the other agenty stuff I've seen it's not clear what the fluff adds over just prompting the base model.
- kosh2 3y agoMaybe simply a larger task? "Write 10 popular mini games" --> What are those games? 1. Agent --> How does each of them work? 2. Agent --> Write each of those. Agent 3-13
- 33a 3y agoI'm just using the examples from the paper. Here's another one: "Implement a Gomoku game using Python, incorporating an AI opponent with varying difficulty levels." One shot prompt works fine: https://chat.openai.com/share/4435152f-ba13-45a5-a116-eb4f57b27ab2 https://chat.openai.com/share/4435152f-ba13-45a5-a116-eb4f57... AFAICT using ChatGPT directly gets you better results with lower cost and less complexity. The "MetaGPT" framework adds bloat and delivers nothing.
- makk 3y agoJust starting to read this. Their Fig. 1 is basically the README of a project we've been quietly working on. It's easy to imagine how systems like these might succeed by following workflows analogous to those humans use, on greenfield toy problems. The trouble is, if you paratrooper something like this into an exiting repository, or you want to build something of real significance, then you've got more context than can fit into an LLMs window. You don't just have the hallucination / stay-on-task problems, you have the problem of the LLM not having everything it needs to know to complete its task. In my experience, the latter problem of ensuring the LLM knows everything it needs to know is the bigger problem. The other stuff (using standard operating procedures and workflows, as in this paper) is actually the relatively easy stuff. I'm not saying it's easy-easy or that it's obvious from the outside if you haven't played around with it; I'm not disparaging the paper. So, what I'm most curious about, and I wonder if anyone here has seen good solutions, is this: How do these multi-agent systems maintain/retrieve the appropriate domain context required to make good design decisions, good coding decisions, and so on? "Use embedding and shove stuff into a vector database" is hand-wavy and doesn't get you all the way there. I'm looking for concrete solutions; academic papers that show a lot of promise; and the like. (Maybe they lay it out in this paper and I just haven't gotten to it yet. But if they've cracked that nut I suspect they would lead with it.)
- WanderPanda 3y agoIs it just me or should it be "Meta Programming for a Collaborative Multi-Agent Framework". I'm not a native speaker but I encounter this kind of weird word order a lot in papers recently and it came to no surprise to me that the author list contains a lot of chinese sounding names. I would be curious if this word order sounds better when translated into Mandarin word by word.
- 33a 3y agoI strongly believe this to be total bullshit. Here's how I managed to recreate their example of a BlackJack game using a single simple prompt in ChatGPT: https://chat.openai.com/share/561419c8-4143-4172-b5e2-a411eeed5c5a https://chat.openai.com/share/561419c8-4143-4172-b5e2-a411ee... I'm not convinced the meta/agenty/company stuff is doing anything to help the LLM generate working code and it looks like they never bothered to check the null hypohesis before hitting publish.
- saliagato 3y agoI ageee. Other examples can be reproduced with a straight input output prompt
- phillipcarter 3y agoMeta-commentary: these different AI agent systems (frameworks?) sure are good at racking up GitHub stars but don't appear to have any use beyond a fun demo. My impression is that, like AutoGPT, this will simply fall over with anything even mildly complicated. Maybe it's the direction we're ultimately headed, but I'm just far too skeptical given what I've seen so far, and I don't believe we just need the right orchestration of models to pull off useful general purpose AI agents.
- penjelly 3y agoagreed. those initial projects didnt go far and they werent particularly innovative imo. Doubt this is any different
- 33a 3y agoI think at least some of it is outright fraud. A lot of these are being pushed by shameless spammers and former crypto hucksters. Some of the vanity metrics like followers and stars can just be bought if you have a few bucks, and boosting those numbers makes it easier to cheat and convince others to give you more money. This cycle effectively pays for itself assuming you don't care about the reputational risks.
- jondwillis 3y ago+1, SuperAGI’s discord feels very crypto.
- cyanydeez 3y agoI'm sensing it's gonna draw the same low code/no code crowd with maybe 5-10% improvement but the same or worse technical debt
- famouswaffles 3y agoIt's more than just a react loop but i think there's promising research in multi gent cooperation. Notice the first doesn't even use GPT-4. https://arxiv.org/abs/2307.07924 https://arxiv.org/abs/2307.07924 https://arxiv.org/abs/2307.02485 https://arxiv.org/abs/2307.02485
- penjelly 3y agostrange to me it was linked here via arxiv instead of github
- jpdus 3y agoI submitted via Github 2 days ago [1] but didn't get any traction. But probably couldn't be submitted again with the Github link... Anyways nice to see some discussion here, didn't read a lot elsewhere about the project. [1] https://news.ycombinator.com/item?id=37053201 https://news.ycombinator.com/item?id=37053201
- Imnimo 3y agoI am very skeptical that doing things like producing "Competitive Quadrant Charts" (see Figure 3) is a helpful form of prompt engineering. The extremely weak ablation (Section 4.4) rings a lot of alarm bells for me also.
- tsunamifury 3y agoWe're doing this with allofus.ai It's pretty interesting how well it works, but there is a lot more to improve as you can refine the agents for more than their generalized domain, into their perspective and real publish knowledge for increased improvement.
- joenot443 3y agoWanted to give it a try, but gating the product behind Twitter/Google SSO was a non-starter for me. Hope you can get a no-login-required demo up eventually!
- tsunamifury 3y agoI think you have to be pretty realistic about the costs of LLM products, there is a reason no one has a non-gated product -- It would be an absolutely ridiculous and uncontrollable cost. That being said, totally understood. Logging in is very little cost though, we just need to make a better value prop upfront.
- joenot443 3y agoThat's true! They're damn expensive.
- primitivesuave 3y agoThis is just a collection of static ChatGPT prompts and an output aggregator - I really don't see any novel research or real-world applicability of this aside from potentially scaffolding a project. One of the assumptions baked into the coding prompt is that you will never need to use an external library or API [1] which is an inherent requirement in most practical use cases. > ## Code: {filename} Write code with triple quoto, based on the following list and context. > 1. Do your best to implement THIS ONLY ONE FILE. ONLY USE EXISTING API. IF NO API, IMPLEMENT IT. 1. https://github.com/geekan/MetaGPT/blob/main/metagpt/actions/write_code.py https://github.com/geekan/MetaGPT/blob/main/metagpt/actions/...
- sprobertson 3y agoThat prompt has a lot of shouting, I kind of feel bad for the prompted.
- I_am_unique6435 3y agoHonestly this paper and many similar are more a scientific proof of what people in the open-source community have already published. Especially after tree-of-thoughts it should be clear to everybody that LLMs profit from the same guidelines and structures that allow humans to think better. The future will likely be simulated companies and the business logic will be how you compose this agent interaction. I haven't run it yet but I feel this specific paper is just good prompt engineering (with a little bit real engineering in it). Also it reads like a pitch honestly. Would bet they are looking for funding rn.