4 ms·
How is that an argument at all? Of course if you could build a better agent that could solve every problem the outcome of the paper would be "this tool performs
by contagiousflow 1y ago
How is that an argument at all? Of course if you could build a better agent that could solve every problem the outcome of the paper would be "this tool performs well at this"
- handfuloflight 1y agoIt is an argument: a flawed agent lead to flawed results. A flawed agent does not speak for all agents.
- oblio 1y agoThe "sufficiently smart compiler" debate, 50 years later :-p
- handfuloflight 1y agohttps://en.wikipedia.org/wiki/Faulty_generalization https://en.wikipedia.org/wiki/Faulty_generalization
- contagiousflow 1y agoBut the argument should be showing an agent that does in fact pass these tests. You can't just assert that "this one failed, but surely there must be some agent that is perfect, therefore you can't generalize".
- handfuloflight 1y agoThat's not my argument. My argument isn't "surely there must be some agent that is perfect", my argument is this test study can't speak for all agents.
- nitwit005 1y agoBut no test can. They ran an experiment, they got this result. You can run more experiments if you want.
- handfuloflight 1y agoI didn't say any test could. I'm pointing out the flaw in the commenters in this thread generalizing the findings.
- notahacker 1y agoEven more so when the context is "this person is an AI research engineer at a company doubling down on AI agents, designing relevant benchmarks and building agents that run on that company's stack" not "this is an AI-skeptic dilettante who wrote a weird prompt". It's not like we have reason to believe the average Salesforce customer is much better at building agents who respect confidence and handle CRM tasks optimally...