3 ms·
The comment you link to says that "we only performed a quick preliminary search" and "We do not have a method for automatically checking existing trajectories."
by comex 1y ago
The comment you link to says that "we only performed a quick preliminary search" and "We do not have a method for automatically checking existing trajectories." In other words, it can't confirm that the issue only "affected a tiny fraction of existing agents in a tiny fraction of their runs" as you say. Are you saying that you have since separately confirmed this?
Edit: That said, I’m willing to believe based on the information in the thread that this most likely only affects a tiny fraction of runs.
- typpilol 1y agoYa what he links directly contradicts what he's saying lol
- ofirpress 1y ago[flagged]
- bflesch 1y agoUnfortunately the bank account trajectories are not public, because unscupulous corporations such FAANG who let thousands of engineers wade through my chat messages on their platforms might not shy away from bribing academics to improve benchmarks of their billion-dollar AI initiatives. It's also a bribe if my sibling gets a job with $500k annual salary. Tech is not immune to it.
- Zacharias030 1y agoYou realize that this problem in SWE-Bench was discovered and publicized by people within those FAANG corporations?
- TheDong 1y agoI'm sure some of the people working at Theranos thought there legitimately was a revolutionary blood-test machine. The presence of a person who wants SWE-bench to have honest results and takes it seriously does not mean the results are free of perverse incentives, nor that everyone is behaving just as honestly.
- Zacharias030 1y agoWhen Swe-Bench was new in 2023, it was — with all due respect — a bit of a niche benchmark in LLM research. LLMs were so incredibly useless at solving these tasks that I think you could find a bit more empathy for the original academic authors. I don’t think the Theranos example applies. Even the flawed benchmark was good enough to get us from ~GPT4 to Claude 4‘s coding ability.
- ares623 1y agoThey really did a "trust me bro" and "do your own research" huh
- stronglikedan 1y agothe strange thing to me is that people would have it any other way. if you don't trust someone, why would you trust them to do the research for you? bit of entitlement if you ask me
- wubrr 1y agoBecause you should never just 'trust' random 'research'. Good analysis in this case will clearly explain the problem, the analysis methodology, findings, net effects, resolution, etc. Something you can read, and decide for yourself whether it is complete/incomplete, has holes, contradictions, etc. Not 'we looked into it and all is good - only potentially tiny effect' (no actual data or methodology presented at all) and then linking to a comment directly contradicting the claim... It's a hilariously unserious and untrustworthy response.
- aprilthird2021 1y agoIt's not that people are entitled. It's that "do your own research" is usually a cop out when you yourself don't understand the answer or are hiding it
- croon 1y agoArguably the initial post was meant to convey confidence and authority on the subject. When questioned you could either dive deeper and explain in more detail why x because of y (if so inclined), ignore it, or... do what they did. No one owes anyone anything, but if you want to represent something; answering the question more in detail would have either closed the issue or raised more scrutiny, both of which are a good thing when trying to figure something out. I don't have to trust someone to check their research and look at how they worked. If the work doesn't pass muster, likely the results don't either. Again, you can view it as entitlement, but if you're not going to bother backing up your claim, why make the claim to start with?
- 1y ago
- deleted 1y ago[deleted]
- plumb_bob_00 1y agoIf you are going to represent your team in public, you owe them better than a response like this.
- keybored 1y agoI say let them cook.
- grepfru_it 1y agoHol up
- hamonrye34 1y agoThis is contingent on whether SWE N-class frontier models can do deep packet inspection.
- phyzome 1y agoThat sounds like the job of the person making the claim.
- typpilol 1y agoAre you saying you've done way more than a cursory search and ruled out everything?