4 ms·

From the website linked in the readme: “A lot of research has been doing in this are and we can expect a lot more in 2024 in this space. I promise to share som

From the website linked in the readme: “A lot of research has been doing in this are and we can expect a lot more in 2024 in this space. I promise to share some clarity around where I think this industry is headed. In personal talks I have warned that multi-agent systems are complex and hard to get right. I've seen little evidence of real-world use cases too” These assistant systems fascinate me, but I just don’t have the time and energy to set something up. I was going to ask if anyone had a good experience with it, but the above makes it sound like there’s not much hope at the moment. Curious what other people’s experience are.

m3kw9 2y ago
By the time you do get around to it OpenAi would have built a full interface for this. This is the type of stuff that’s gonna get steamrolled.
- WXLCKNO 2y ago
  Pretty much this. I'd love counter examples of startups in the space that haven't been crushed from the top yet.
- mokanfar 2y ago
  I'm impatiently waiting to become the ultimate armchair music video director I've always dreamed of once this video AI thing rolls out...
metaskills 2y ago
Thanks @beoberha, I am too. I like one take I heard on Twitter. The sentiment was something like these types of systems are useful under the AI-Powered Productivity industry which has incremental gains, no big bangs. Said another way, if your job was to help a TON of your employees be more productive individually, it is worth it because companies measure those efforts broadly and the payoff is there. But again, not big. My advice for folks to stay lower level and hook AI automation up with simple, closed loop, LLM patterns that feel more like basic API calls in a choreographed manner. OMG, hope all that made sense
- purposesystem 2y ago
  that's actually a great reply, thanks
dongobread 2y ago
We tried using a multi-agent system for a complex NLP-type task and we found: - Too many errors that just propogate on top of each other, if a single agent in the chain generates something even a little bit off then the whole system goes off the rails. - You often end up having to pass a massive amount of shared context to every agent which just increases the cost dramatically. Curiously enough we had an architect from OpenAI tell us the same thing about agent systems a few days ago (our company is a big spender so they serve a consulting function), so I don't think anybody is really finding success with multi-agent systems currently. IMO the core tech is nowhere near good enough yet.
- pennomi 2y ago
  > Too many errors that just propogate on top of each other LLMs are like the perfect improv comedy troupe, they virtually always say “yes, and…”
  - echelon 2y ago
    > perfect improv comedy troupe Check out Vtubers like CodeMiko, who improvs against LLM agents. Or 24/7 streaming LLM cartoon shows that take audience plot suggestions.
- lmeyerov 2y ago
  we do multistep programs in louie.ai via a variety of agents/tools, like "get X data from DB Y, wrangle cols A+B in Python, and then draw an interactive map + graph" The ultimate answer is fairly short if you are a senior python data scientist, like 50loc. The agents will wander and iterate until they push through. You might correct & tweak if a bit off. Importantly, this does agents opposite of the way Devin AI engineer replacements are presented. Here, you get it to do a few steps, and then move on to the next few steps. The agents still crank away a ton and do all sorts of clever things for you... to get you more reliably to the next step, vs something big & wrong.
  - arresin 2y ago
    So the human is like a reviewer, coming in, checking things, tweaking etc, then sending it back to the machine? (At which point the cycle continues)
fzliu 2y ago
A lot of folks I've spoken with say that single-agent systems are still extremely limited, let alone multi-agent platforms. In general, it seems to boil down to: - Agents need lots of manual tuning and guardrails to make them useful - Agents with too many guardrails are not general-purpose enough to be worth the time and effort to build I believe truly great agents will only come from models whose weights are dynamically updated. I hope I'm wrong.
deleted 2y ago
[deleted]

m3kw9 2y ago

By the time you do get around to it OpenAi would have built a full interface for this. This is the type of stuff that’s gonna get steamrolled.

WXLCKNO 2y ago
Pretty much this. I'd love counter examples of startups in the space that haven't been crushed from the top yet.
mokanfar 2y ago
I'm impatiently waiting to become the ultimate armchair music video director I've always dreamed of once this video AI thing rolls out...

metaskills 2y ago

Thanks @beoberha, I am too. I like one take I heard on Twitter. The sentiment was something like these types of systems are useful under the AI-Powered Productivity industry which has incremental gains, no big bangs. Said another way, if your job was to help a TON of your employees be more productive individually, it is worth it because companies measure those efforts broadly and the payoff is there. But again, not big. My advice for folks to stay lower level and hook AI automation up with simple, closed loop, LLM patterns that feel more like basic API calls in a choreographed manner. OMG, hope all that made sense

purposesystem 2y ago
that's actually a great reply, thanks

dongobread 2y ago

We tried using a multi-agent system for a complex NLP-type task and we found: - Too many errors that just propogate on top of each other, if a single agent in the chain generates something even a little bit off then the whole system goes off the rails. - You often end up having to pass a massive amount of shared context to every agent which just increases the cost dramatically. Curiously enough we had an architect from OpenAI tell us the same thing about agent systems a few days ago (our company is a big spender so they serve a consulting function), so I don't think anybody is really finding success with multi-agent systems currently. IMO the core tech is nowhere near good enough yet.

pennomi 2y ago
> Too many errors that just propogate on top of each other LLMs are like the perfect improv comedy troupe, they virtually always say “yes, and…”
- echelon 2y ago
  > perfect improv comedy troupe Check out Vtubers like CodeMiko, who improvs against LLM agents. Or 24/7 streaming LLM cartoon shows that take audience plot suggestions.
lmeyerov 2y ago
we do multistep programs in louie.ai via a variety of agents/tools, like "get X data from DB Y, wrangle cols A+B in Python, and then draw an interactive map + graph" The ultimate answer is fairly short if you are a senior python data scientist, like 50loc. The agents will wander and iterate until they push through. You might correct & tweak if a bit off. Importantly, this does agents opposite of the way Devin AI engineer replacements are presented. Here, you get it to do a few steps, and then move on to the next few steps. The agents still crank away a ton and do all sorts of clever things for you... to get you more reliably to the next step, vs something big & wrong.
- arresin 2y ago
  So the human is like a reviewer, coming in, checking things, tweaking etc, then sending it back to the machine? (At which point the cycle continues)

fzliu 2y ago

A lot of folks I've spoken with say that single-agent systems are still extremely limited, let alone multi-agent platforms. In general, it seems to boil down to: - Agents need lots of manual tuning and guardrails to make them useful - Agents with too many guardrails are not general-purpose enough to be worth the time and effort to build I believe truly great agents will only come from models whose weights are dynamically updated. I hope I'm wrong.

deleted 2y ago

[deleted]