5 ms·

How to enhance generative AI's problem-solving capabilities, boost productivity

40 points by monkeydust 2y ago

smarm52 2y ago
A nice summary of an idea similar to Multi-Agent Systems. https://en.wikipedia.org/wiki/Multi-agent_system https://en.wikipedia.org/wiki/Multi-agent_system
NicoJuicy 2y ago
These things will have the same flaw as no-code tools. No one is going to remember how the system works and all those prompt engineers are going to find out that programming languages are well documented, but things like migrations, multitenancy, ... aren't. Good luck when an AI api implements a breaking change in an API and people rely on it. Or when a issue happens and it can't find logs, ... ( If it was even implemented :p )
Animats 2y ago
The article starts out as if it's headed for "and that's how we did it". But no. There's no implementation. "Imagine a virtual team of AI agents, each with its workflow’s own specialism, collaborating to solve problems and make decisions just like a human team would." OK. Where does that go? So far, multi-agent systems have been delegating simple and well-bounded tasks, such as "fetch the weather info for Outer Nowhere" or "check airline schedules for flights from JFK to ORD", or even "what is 25% of $50". Those are questions inexpensive to answer, and don't need much management. If the subagents are complex, they will need management, and probably budgeting. Subagents need to know when to stop and when to approximate. If the subagents are themselves generative AI systems, there's potential for hallucination at the lower levels generating info that the higher levels take as valid. Subagents also need to be able to query their managers - "is this enough detail" is a reasonable question to pass upwards. They may need to talk to their peer agents. Now you have all the problems of organizational dynamics within a multi-agent AI system. I look forward to reading papers with titles such as: - "Teams of generative AI agents for coding - scrum or waterfall?" - "Span of control - how many subagents should an agent manage?" - "Does the agent org chart influence the solution too much?" - "Resolving disagreements between specialized subagents". That's where this is going. It has to. Once you start to cut a problem into pieces to be handled by different units, all those problems arise.
- fsndz 2y ago
  this is why I have been arguing there will be no AGI https://www.lycee.ai/blog/why-no-agi-openai https://www.lycee.ai/blog/why-no-agi-openai
  - prox 2y ago
    Its a goal looking for a problem, aka AI hype. It’s the big thing now, like NFTs before. I agree with your blog, the definition itself is vague, and what people want to get out of it as well. There is a big “we’ll figure it out when we get there” attitude it seems to me. Imagine a society where everything can be and is done by AGI (and its drones) what then? What do we want out of it? What will define humans in such an environment?
    - fsndz 2y ago
      can't agree more !
    - ben_jones 2y ago
      People push back on the Crypto/NFT <—> LLMs comparison but I think it’s spot on. Crypto failed because it could never be honest about its trade offs and the reasons for centralization in the first place. It couldn’t be honest about its flaws because then VCs and private equity would make less money. LLMs as an immediate panacea is failing because it can’t be honest about what actual intelligence is, where human-computer-interaction is, and its ultimate goal of culling jobs. It couldn’t be honest about its flaws because then VCs and private equity would make less money. If only there was some pattern involved here we could avoid the pitfalls of these hype cycles - the pitfalls that end up in a whole lot of people being worse off and a couple new Lake Tahoe vacation home purchases.
      - prox 2y ago
        I think you nail it. It is all about that honesty vs the hype.
  - namaria 2y ago
schmidtleonard 2y ago
> The productivity benefits perhaps take us closer to the aspiration Keynes had when he wrote Economic Possibilities for our Grandchildren in 1930, in which he forecast that in a hundred years, thanks to technological advancements improving the standard of living, we could all be doing 15-hour work weeks. Well, you see, the benefits have to be split between capital and labor. The system is called "capitalism." Figure it out.
- ben_w 2y ago
  We have a higher standard of living than when the book was written. Even the basics have changed a lot: > It wasn't until the 1930s that new houses were built with indoor toilets and bathrooms as standard, says Zoe Hendon, head of museum collections at Middlesex University's Museum of Domestic Design and Architecture. "At that time, bathrooms were seen as a luxury." - https://www.bbc.com/culture/article/20210407-how-the-bathroom-became-the-ultimate-sanctuary https://www.bbc.com/culture/article/20210407-how-the-bathroo...
  - rakoo 2y ago
    Capitalism isn't about standards of livings, it's about exploitation of the majority by and for a minority, whether you give the exploited shackles or golden shoes.
    - prox 2y ago
      Yup, if you read the major economic books coming out of the last decade, they all have that theme.
    - ben_w 2y ago
      Capitalism, The Wealth of Nations, the observation that people making the best choices for themselves is often good for the collective benefit of society, because this reduces waste. A century later, communism, The Communist Manifesto, was the observation that in practice capitalism is, much like its various predecessors, yet another way for minorities to rule over the masses. Unfortunately, the utopian attempts to replace capitalism with communism have thus far demonstrated exactly the same problems that capitalism has. A century later, we got Nash game theory, formalising the tragedy of the commons, the prisoner's dilemma, etc., — I suspect "the next big thing" will be based on this. (Scare quotes because it may already exist: communist thought had precursors before the Manifesto, and there was a big gap between the publication of the Manifesto and the Russian revolution).
hubraumhugo 2y ago
I recently wrote a blog post about why "AI agents" are still too early, too expensive, too unreliable: https://www.kadoa.com/blog/ai-agents-hype-vs-reality https://www.kadoa.com/blog/ai-agents-hype-vs-reality The WebArena leaderboard[0], which benchmarks LLM agents against real-world tasks, shows that even the best-performing models have a success rate of only 35.8%. [0] https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC_pF7LdyGU1f_ufZb_NWNBZQ/edit#gid=0 https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC...
- 3abiton 2y ago
  I'm curious about the limitation? Wouldn't a team of AI agents that is well designed complete such tasks? I assume the challenge would be in coordination and autonomy?
  - refulgentis 2y ago
    All of these require manipulating browser DOM/vision, best one is at sub 40%. (On mobile, will take too long, but I'll check later if anyone wants a paper link)
chx 2y ago
> How to enhance generative AI's problem-solving capabilities, A zero multiplied by whatever is still zero. It can not solve anything with one broad category of exceptions as https://hachyderm.io/@inthehands/112006855076082650 https://hachyderm.io/@inthehands/112006855076082650 brilliantly explains: > You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to. > Alas, that does not remotely resemble how people are pitching this technology.
- refulgentis 2y ago
  This assumes its consequent, and that's it. I think we're all against wrong AI hype. Vast assertions like "(they) claim AI replaces thought" require a wee bit more fleshing out, too abstract.

40 points by monkeydust 2y ago

A nice summary of an idea similar to Multi-Agent Systems. https://en.wikipedia.org/wiki/Multi-agent_system https://en.wikipedia.org/wiki/Multi-agent_system

NicoJuicy 2y ago

These things will have the same flaw as no-code tools. No one is going to remember how the system works and all those prompt engineers are going to find out that programming languages are well documented, but things like migrations, multitenancy, ... aren't. Good luck when an AI api implements a breaking change in an API and people rely on it. Or when a issue happens and it can't find logs, ... ( If it was even implemented :p )

Animats 2y ago

The article starts out as if it's headed for "and that's how we did it". But no. There's no implementation. "Imagine a virtual team of AI agents, each with its workflow’s own specialism, collaborating to solve problems and make decisions just like a human team would." OK. Where does that go? So far, multi-agent systems have been delegating simple and well-bounded tasks, such as "fetch the weather info for Outer Nowhere" or "check airline schedules for flights from JFK to ORD", or even "what is 25% of $50". Those are questions inexpensive to answer, and don't need much management. If the subagents are complex, they will need management, and probably budgeting. Subagents need to know when to stop and when to approximate. If the subagents are themselves generative AI systems, there's potential for hallucination at the lower levels generating info that the higher levels take as valid. Subagents also need to be able to query their managers - "is this enough detail" is a reasonable question to pass upwards. They may need to talk to their peer agents. Now you have all the problems of organizational dynamics within a multi-agent AI system. I look forward to reading papers with titles such as: - "Teams of generative AI agents for coding - scrum or waterfall?" - "Span of control - how many subagents should an agent manage?" - "Does the agent org chart influence the solution too much?" - "Resolving disagreements between specialized subagents". That's where this is going. It has to. Once you start to cut a problem into pieces to be handled by different units, all those problems arise.

fsndz 2y ago
this is why I have been arguing there will be no AGI https://www.lycee.ai/blog/why-no-agi-openai https://www.lycee.ai/blog/why-no-agi-openai
- prox 2y ago
  Its a goal looking for a problem, aka AI hype. It’s the big thing now, like NFTs before. I agree with your blog, the definition itself is vague, and what people want to get out of it as well. There is a big “we’ll figure it out when we get there” attitude it seems to me. Imagine a society where everything can be and is done by AGI (and its drones) what then? What do we want out of it? What will define humans in such an environment?
  - fsndz 2y ago
    can't agree more !
  - ben_jones 2y ago
    People push back on the Crypto/NFT <—> LLMs comparison but I think it’s spot on. Crypto failed because it could never be honest about its trade offs and the reasons for centralization in the first place. It couldn’t be honest about its flaws because then VCs and private equity would make less money. LLMs as an immediate panacea is failing because it can’t be honest about what actual intelligence is, where human-computer-interaction is, and its ultimate goal of culling jobs. It couldn’t be honest about its flaws because then VCs and private equity would make less money. If only there was some pattern involved here we could avoid the pitfalls of these hype cycles - the pitfalls that end up in a whole lot of people being worse off and a couple new Lake Tahoe vacation home purchases.
    - prox 2y ago
      I think you nail it. It is all about that honesty vs the hype.
- namaria 2y ago

schmidtleonard 2y ago

> The productivity benefits perhaps take us closer to the aspiration Keynes had when he wrote Economic Possibilities for our Grandchildren in 1930, in which he forecast that in a hundred years, thanks to technological advancements improving the standard of living, we could all be doing 15-hour work weeks. Well, you see, the benefits have to be split between capital and labor. The system is called "capitalism." Figure it out.

ben_w 2y ago
We have a higher standard of living than when the book was written. Even the basics have changed a lot: > It wasn't until the 1930s that new houses were built with indoor toilets and bathrooms as standard, says Zoe Hendon, head of museum collections at Middlesex University's Museum of Domestic Design and Architecture. "At that time, bathrooms were seen as a luxury." - https://www.bbc.com/culture/article/20210407-how-the-bathroom-became-the-ultimate-sanctuary https://www.bbc.com/culture/article/20210407-how-the-bathroo...
- rakoo 2y ago
  Capitalism isn't about standards of livings, it's about exploitation of the majority by and for a minority, whether you give the exploited shackles or golden shoes.
  - prox 2y ago
    Yup, if you read the major economic books coming out of the last decade, they all have that theme.
  - ben_w 2y ago
    Capitalism, The Wealth of Nations, the observation that people making the best choices for themselves is often good for the collective benefit of society, because this reduces waste. A century later, communism, The Communist Manifesto, was the observation that in practice capitalism is, much like its various predecessors, yet another way for minorities to rule over the masses. Unfortunately, the utopian attempts to replace capitalism with communism have thus far demonstrated exactly the same problems that capitalism has. A century later, we got Nash game theory, formalising the tragedy of the commons, the prisoner's dilemma, etc., — I suspect "the next big thing" will be based on this. (Scare quotes because it may already exist: communist thought had precursors before the Manifesto, and there was a big gap between the publication of the Manifesto and the Russian revolution).

hubraumhugo 2y ago

I recently wrote a blog post about why "AI agents" are still too early, too expensive, too unreliable: https://www.kadoa.com/blog/ai-agents-hype-vs-reality https://www.kadoa.com/blog/ai-agents-hype-vs-reality The WebArena leaderboard[0], which benchmarks LLM agents against real-world tasks, shows that even the best-performing models have a success rate of only 35.8%. [0] https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC_pF7LdyGU1f_ufZb_NWNBZQ/edit#gid=0 https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC...

3abiton 2y ago
I'm curious about the limitation? Wouldn't a team of AI agents that is well designed complete such tasks? I assume the challenge would be in coordination and autonomy?
- refulgentis 2y ago
  All of these require manipulating browser DOM/vision, best one is at sub 40%. (On mobile, will take too long, but I'll check later if anyone wants a paper link)

chx 2y ago

> How to enhance generative AI's problem-solving capabilities, A zero multiplied by whatever is still zero. It can not solve anything with one broad category of exceptions as https://hachyderm.io/@inthehands/112006855076082650 https://hachyderm.io/@inthehands/112006855076082650 brilliantly explains: > You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to. > Alas, that does not remotely resemble how people are pitching this technology.

refulgentis 2y ago
This assumes its consequent, and that's it. I think we're all against wrong AI hype. Vast assertions like "(they) claim AI replaces thought" require a wee bit more fleshing out, too abstract.