10 ms·
Killed by LLM
- deleted 2y ago[deleted]
- matt3210 2y agoI read recently that small variations in the tests cause failures by large margins. If this doesn’t show over fitting in don’t know what would.
- lxgr 2y agoWasn’t that for human tests, i.e. not specifically AI benchmarks? Benchmarks should generally not be game-able by overfitting.
- matt3210 2y agoThe article shows all the tests against human performance. The math one in particular is the one where small variations reduce the success rate significantly. I can’t find the source but it was pasted here in the last 2 weeks.
- CrazyStat 2y agoYou're probably remembering this: https://news.ycombinator.com/item?id=42565606 https://news.ycombinator.com/item?id=42565606
- friend_Fernando 2y agoEventually, all the better AGI tests should have large private evaluation datasets with no possible cheating or feedback loops. We're getting there.
- tharkun__ 2y agoHow does this site make sense? It lists the "Turing test" as "original" at greater than 50% and the the AI that "beat" it at 46%. At that point I just stopped scrolling.
- mattnewton 2y agothe Turing test is scored by how often an interrogator can determine if they are talking to a machine or a human. It’s perhaps a confusing way to show it, and leaves out a lot of important information about the result they are citing, but they are saying before the interrogator did better than chance and after gpt the interrogator guesses right slightly worse than chance.
- lxgr 2y agoPresumably it means that the human detected the AI correctly less than 50% of the time, averaged over a repeated number of experiments.
- bacheaul 2y agoI read that as the pass mark being able to identify human vs machine more than 50% of the time. At 50% it's no better than randomly guessing.
- deleted 2y ago[deleted]
- meltyness 2y agoI wonder if there's a hyper-Turing test where an AI passes if the model, itself, cannot determine if it is talking to itself; or perhaps stated differently, maximizing some measure of control and processing duration to successfully conceal its identity under forced processing, discounting a solution that specifically learns to be silent or incoherent. I'm not sure what the value would be, just a passing thought. This is probably already happening within the parade of censorship systems trying to imbue the models with agency
- junon 2y agoScore is based on the interrogator, a human. If you read a Markov chain bot's text you'd guess it was a bot probably 80-100% of the time. With a real human, you'd guess it was a bot maybe 0-30% of the time, depending. I'm making up these figures, but the point is lower is better, or "more Human-Like". Test was specified as >50% meaning "accurately determined human vs. bot more than half the time". The site claims LLMs are now guessed correctly less than half, which is how the turing test was defined as per the site. It makes sense, even if you disagree it's significant.
- dleavitt 2y agoThe layer with the radial gradient you're putting in front of the Turing Test card blocks interaction with it - can't click or hover on its links.
- krackers 2y agoEverything says "killed by saturation". Is there another way to be killed?
- varelse 2y ago[dead]
- bufferoverflow 2y agoThere are benchmarks that humans score close to zero on average and the top LLM scores 25%. https://epoch.ai/frontiermath/the-benchmark https://epoch.ai/frontiermath/the-benchmark
- pama 2y agoIf anyone from epoch.ai is reading this, it would be nice to link the toplevel result for o3 to this page.
- mrayycombi 2y agoBragging about how LLMs defeated maginot line defenses that can be trained around, makes us feel warm and fuzzy. Too bad the real world isn't like that.
- solarkraft 2y agoThe page doesn’t seem to define what „killed“ or „defeated“ means. The LLM being better than a human? The LLM having been trained against the benchmark, making it useless?
- anonymoushn 2y agoIt does if you scroll down.
- lamename 2y agoPosted by Chollet himself: > I don't think people really appreciate how simple ARC-AGI-1 was, and what solving it really means. It was designed as the simplest, most basic assessment of fluid intelligence possible. Failure to pass signifies a near-total inability to adapt or problem-solve in unfamiliar situations. > Passing it means your system exhibits non-zero fluid intelligence -- you're finally looking at something that isn't pure memorized skill. But it says rather little about how intelligent your system is, or how close to human intelligence it is. https://bsky.app/profile/fchollet.bsky.social/post/3les3izgdj22j https://bsky.app/profile/fchollet.bsky.social/post/3les3izgd...
- energy123 2y ago> Failure to pass signifies a near-total inability to adapt or problem-solve in unfamiliar situations. Not necessarily. Get a human to solve ARC-AGI if the problems are shown as a string. They'll perform badly. But that doesn't mean that humans can't reason. It means that human reasoning doesn't have access to the non-reasoning building blocks it needs (things like concepts, words, or in this case: spatially local and useful visual representations). Humans have good resolution-invariant visual perception. For example, take an ARC-AGI problem, and for each square, duplicate it a few times, increasing its resolution from X*X to 2X*2X. To a human, the problem will be almost exactly equally difficulty. Not for LLMs that have to deal with 4x as much context. Maybe for an LLM if it can somehow reason over the output of a CNN, and if it was trained to do that like how humans are built to do that.
- refulgentis 2y agoExcellent point, I'm not sure people are aware, but these are straight-up lifted from standard IQ tests, so they're definitely not all trivially humanly solvable. I needed an official one for medical reasons a few years back
- echelon 2y agoARC-AGI feels like it would fall to a higher dimensional convolution rather than reasoning.
- 2y ago
- 0xDEAFBEAD 2y agoI assumed this was about chatbot users committing suicide in order to "join" the bot they are chatting with. It's already happened a couple of times, apparently: https://futurism.com/teen-suicide-obsessed-ai-chatbot https://futurism.com/teen-suicide-obsessed-ai-chatbot https://garymarcus.substack.com/p/the-first-known-chatbot-associated https://garymarcus.substack.com/p/the-first-known-chatbot-as...
- sam0x17 2y agoSimilarly I thought it would be about ML and data projects that have become defunct due to the advent of LLMs.
- nayuki 2y agoI thought the title meant that a chatbot gave bad medical, engineering, and/or safety-critical advice that a human ended up following.
- cdev_gl 2y agoYea, I too was not expecting a list of past benchmarks. If not the aforementioned actual human deaths, I had expected either a list of companies whose pivot to AI/LLMs led to their downfall (but I guess we're going to need to wait a year or two for that) or a list of industries (such as audio transcription) that are being killed by AI as we speak. We really do live in interesting times. Usually I feel pretty confident about predicting how a trend will continue, but as it is the only prediction I can make with confidence for this latest AI research is that it is and will be used by militaries to kill a lot of people. Oh, hey, that's another thing this article could have listed! Outside of that, all bets are open. Possible wagers include: "Turns out to be mostly useful in specific niche applications and only seemingly useful anywhere else", "Extremely useful for businesses looking to offset responsibility for unpopular decisions", "Ushers in an end to work and a golden age for all mankind", "Ushers in an end to work and a dark age for most of the world", "Combines with profit motives to damage all art, culture, and community", etc etc. I know many folk have strong opinions one way or the other, but I think it's literally anyone's game at this point, though I will say I'm not leaning optimistic.
- rasz 2y ago
- anonymoushn 2y agoInteresting choice having a little (i) icon in the Turing Test card but having mouseover not bring up any text. Or having the link icons in that card that you can click on to do nothing.
- fenomas 2y agoLooks like a bug - that card has an overlay at a higher z-index that obscures its mouseover and clicks. In the source the (i) links to Turing's original "Imitation Game" paper, and the (?) has this hover text: > (?) While the Turing Test remains philosophically significant, modern LLMs can consistently pass it, making it no longer effective at measuring the frontier of AI capabilities.
- levocardia 2y agoI don't really understand why "Killed by: Saturation" is needed - what other options are there? It would also be nice to see the "unbeaten" list: standardized tests LLMs still fail (for now). e.g. Wozniak's coffee test.
- themanmaran 2y agoWozniak's coffee test would be a really fun one to attempt. As long as you could get a capable enough robot, I imagine it's possible. Something like the Spot Arm[1] would be sufficient. Something like: - Key the robot controls to a series of tools (move_forward(x), extend_arm(y)) - Add a camera and pass each frame to the AI model along with the task "make a cup of coffee" and the list of available tools it can call. And it would likely succeed some percentage of the time today! [1] https://bostondynamics.com/products/spot/arm/ https://bostondynamics.com/products/spot/arm/
- ultrablack 2y agoThe tortoise lays on its back, its belly baking in the hot sun, beating its legs trying to turn itself over, but it can't. Not without your help. But you're not helping.
- knowaveragejoe 2y agoI didn't know ARC-AGI had been "beaten" by o3. What are the next challenges that frontier models like o1/o3 are faced with?
- chriscappuccio 2y agoo1 did terrible. o3 did well on arc-agi-pub (public training data) but hasn't passed the private test yet.
- lucianbr 2y agoIs the test still private once it has been run? If you call the OpenAI API and send it some data, OpenAI has access to the data. Did the benchmaker run the models locally somehow?
- vbarrielle 2y agoThe private test is supposed to be run by the ARC-AGI organization themselves, without network access. That's why o3 has not been run against it yet. Not sure if it will be possible either, depends on what OpenAI is prepared to do about it.
- sinuhe69 2y agoI find that MATH challenge “solved” by AI hard to believe. The reason given was “saturation”. Could anybody help explain it a bit? Also in my daily encounter, I stop find a lot of simple math problems all the frontier models could not solve: long logic puzzle, many cases reasoning, and particularly geometry problems. I don’t know where the 97% number for o1 does come from, but in my experience they are much lower than that and math, even elementary maths certainly can not be considered to be “solved”. As far as I can see, OpenAI has been trained their models on all these public problems, so testing on them to record a benchmark is tainted as best when not outright cheating.
- Taek 2y agoI've found o1 to be entirely useful at math problems that are beyond my own (admittedly modest) skills. I've had it write full proofs of correctness for me (one shot, verified), I've had it optimize equations to reduce necessary precision, I've had it optimize equations to remove specific expensive operations (making them computationally more efficient), and finally I've had it prove a handful of my conjectures, which was helpful for taking algorithmic shortcuts in a security sensitive environment. Mostly all algebra and calculus, but definitely all problems that most undergrads would struggle with. It's most useful because it has deep knowledge of related and adjacent conjectures that are well understood, even if you've never heard of them. So it can mix and match things with a lot more ease than a tinkering mathematician
- hgomersall 2y agoYeah, I had (so far apparent, but still be verified) success with o1 teaching me the necessary physics and maths I need to solve my specific problem. This is definitely grad level stuff but well understood. My concern though is it's missing things that are more esoteric.
- gazchop 2y agoIt can't handle trigonometric identities and any form of calculus at the same time without fucking it up. Also abstract stuff like symmetry groups, nope! And anything which involves vectors is a mess. The big problem is it confidently answers the questions utterly wrongly. This is stuff I expect a basic mathematics undergrad to be able to work out in their first or second year.
- Ukv 2y agoIMO a critical feature of the Turing test/imitation game, which many modern implementations including this site's linked paper ignore, is that the interrogator talks to both a human and a bot and must decide that one xor the other is a human. So fooling an interrogator means having them choose the bot as human over an actual human, not just judging the bot to be human (while probably judging humans to be human even more frequently). When the interrogator is only answering "do you think your conversation partner was a human?" individually, bots can score fairly highly simply by giving little information in either direction - like pretending to be a non-english-speaking child, or sending very few messages. Whereas when pitted against a human, the bot is forced to give stronger or equally strong evidence of being human as the average human (over enough tests). To be chosen as human, giving 0 evidence becomes a bad strategy when the opponent (the real human) is likely giving some positive non-zero evidence towards their personhood.
- silisili 2y agoI'm skeptical on the claim. I think most folks, given the test you describe, would be able to pick out which is human. I think it can get there, but I'm not sure anyone has made one yet. ChatGPT responses are heavily downvoted and mocked because they're easy to spot. Does there exist a public LLM that isn't so...wordy, excited, and guardrailed all the time? You can pretty much spot the bot today by prompting something horribly offensive. Their response is always very inhuman, probably due to lack of emotional energy.
- seanmcdirmid 2y agoAren't you just describing those emails in a big corp that are supposedly still written by humans? Yes, they are wordy, excited, and guardrailed, but I don't think they are written by AI yet. I guess this is why LLMs are so feared by high school English teachers. Yes, they don't write well, but neither do their students.
- brabel 2y agoNowadays, it's usually the opposite: if the text is too good, everyone starts accusing it of being written by AI.
- benreesman 2y agoThis technology is useful and interesting and even fun in spite of the ugliest broad-based cash and power grab since 1999. When this godawful once in a generation hype cycle dies down this stuff is going to be strictly awesome.
- j45 2y agoI'm not sure if LLMs have beaten the standards, as much have the information to reply to them as needed. Last week there was a post where slightly changing one of the tests caused LLMs to drop off drastically.
- deleted 2y ago[deleted]
- erichocean 2y agoI'm working on operationalizing AI, and our Turing test is if—by watching a screenshare of the AI worker—you can tell an AI worker (vs. a human) did the task. If you can't, the AI worker passes the test.
- yamrzou 2y agoARC-AGI is not yet killed by LLM. O3 achieved a breakthrough only on ARC-AGI-PUB, which is semi-private. Nothing guarantees that the test data wasn't leaked to OpenAI in previous testing rounds, because the model is not running offline. See: https://news.ycombinator.com/item?id=42478098 https://news.ycombinator.com/item?id=42478098
- anon373839 2y agoI think this should be discussed more. Models that can only be accessed via API cannot be tested without giving their owners access to the test data. You just have to trust that they’ll do the right thing.
- Tepix 2y agoIn particular, in cases where the model gets 16 hours to solve a task that a human can solve in a few minutes, cheating is trivial!
- Tepix 2y agoSee https://arcprize.org/blog/oai-o3-pub-breakthrough https://arcprize.org/blog/oai-o3-pub-breakthrough ARC-AGI-1 will be replaced by ARC-AGI-2 So yes, ARC-AGI-1 was killed.
- deleted 2y ago[deleted]
- alganet 2y agoAn very reliable, very unethical test would be to deploy LLMs on the internet as humans and gauge how other humans react (ignore, call out as LLM, engage, etc). There isn't much in the way of stopping a company from doing that (there should be!).