4 ms·
I recently wrote a blog post about why "AI agents" are still too early, too expensive, too unreliable: https://www.kadoa.com/blog/ai-agents-hype-vs-reality http
by hubraumhugo 2y ago
I recently wrote a blog post about why "AI agents" are still too early, too expensive, too unreliable: https://www.kadoa.com/blog/ai-agents-hype-vs-reality https://www.kadoa.com/blog/ai-agents-hype-vs-reality
The WebArena leaderboard[0], which benchmarks LLM agents against real-world tasks, shows that even the best-performing models have a success rate of only 35.8%.
[0] https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC_pF7LdyGU1f_ufZb_NWNBZQ/edit#gid=0 https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC...
- 3abiton 2y agoI'm curious about the limitation? Wouldn't a team of AI agents that is well designed complete such tasks? I assume the challenge would be in coordination and autonomy?
- refulgentis 2y agoAll of these require manipulating browser DOM/vision, best one is at sub 40%. (On mobile, will take too long, but I'll check later if anyone wants a paper link)