12 ms·
IQ tests results for AI
- pdhborges 1y agoIf the AI is so smart why are we feeding so many dumb humans?
- amunozo 1y agoBabe wake up. New benchmark to overfit models just dropped.
- testdelacc1 1y agoThey’re definitely going to overfit on this, but this will be much better from a marketing perspective. Normies don’t know wtf an MMLU is, but they do know what IQ is and that 140 is a big number. Can’t wait for CEOs to start saying “why would we hire a 120 IQ person who works 9-5 with a lunch break when we can hire a 170 IQ worker who works 24x7 for half the cost??”
- codr7 1y agoYou won't have to, it's already happening.
- mirekrusin 1y agoLet's wait till AI makes hiring decisions.
- bitwize 1y agoYou won't have to, it's already happening.
- mirekrusin 1y agoIs it discriminating towards life forms?
- terminalshort 1y ago"None of the human candidates were qualified. You should hire more computers."
- notahacker 1y ago"Workers rejoice as model overfitted to score 170 on IQ test turns out to be incapable of performing basic tasks..."
- CamperBob2 1y agoNothing matters except the first couple of time derivatives. The workers aren't getting any better.
- kcplate 1y agoThere are a lot of people with high IQs that appear to be incapable of performing basic tasks too
- notahacker 1y agoWell yeah, I'd be wary of an "all our candidates are MENSA members" staffing agency as well :). But human intelligence isn't quite so easily overfitted to a training set consisting of multiple choice matrices and logic puzzles as neural networks
- kcplate 1y agoSure, I’d probably go the extra step of sorting out any applicant who advertised their MENSA membership…but that’s just me finding those folks pretty insufferable. But back to the point it’s really more about if the tool for the job can’t do the job…they are the wrong tool for the job. Tool of course being device, AI, or…human.
- perching_aix 1y agoI sure hope they do, will finally give people some sense of just how little IQ tests actually say. Which mirrors my general experience with automation and AI. Once something is automated and/or AI can do it, the magic goes away, and we're that much ahead in peeling back what is it that actually differentiates us and how exactly.
- scotty79 1y agoThey have offline test that's supposedly not in the training data. It gets lower scores but best one is still 120 IQ.
- dang 1y agoCan you please make your substantive points thoughtfully and avoid the snarky one liners here, tempting though those of course are? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- jonplackett 1y agoReally need to use a CDN before you get #1 on HN
- ranguna 1y agoIirc HN itself does not have a CDN.
- kator 1y agoLLM vibe coded site and architecture?
- scotty79 1y agoDumping things on Cloudflare is clever architecture now?
- mirekrusin 1y agoHow many microservices, sql joins, distributed kafka piplelines etc. we currently recommend for serving static, public article?
- diggan 1y agoI mean not really, as always you just need to make sure you're not doing 10s of dynamic calls for each page load and if you do, add some minute-long cache at least. Most of the stuff that gets hugged to death really shouldn't, most of the times it's just static content that is trivial to host on even $10/month instances.
- FranOntanaya 1y agoThe amount of calls on some pages displaying the simplest stuff is mind-boggling. 160 requests for a page just displaying a HTML5 video and a title, 360 requests for a Reddit page, it's nuts. We don't need to be like this.
- yetihehe 1y ago"We and our 350 partners care about your privacy".
- gpt5 1y agoThe way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others - the “positive manifold.” They then hypothesized a general factor, “g,” to explain this pattern. Early tests (e.g., Binet–Simon; later Stanford–Binet and Wechsler) sampled a wide range of tasks, and researchers used correlations and factor analysis to extract the common component, then norm it around 100 with a SD of 15 and call it IQ. IQ tend to meaningfully predicts performance across some domains especially education and work, and shows high test–retest stability from late adolescence through adulthood. It is also tend to be consistent between high quality tests, despite a wide variety of testing methods. It looks like this site just uses human rated public IQ tests. But it would have been more interesting if an IQ test was developed specifically for AI. I.e. a test that would aim to Factor out the strength of a model general cognitive ability across a wide variety of tasks. It is probably doable by doing principal component analysis on a large set of benchmarks available today.
- anywhichway 1y agoOne potential issue with that approach is the factors wouldn't stay very constant across generations of AI models. While a lot of people have used various methods to try to gauge the strength of various AI models, one of my favorites is this time horizon analysis [1] which took coding tasks of various lengths and looked at how long it takes to humans to complete those tasks and compared that to chance that the AI would successfully complete the task. Then they looked at various threshholds to see how long of tasks an AI could generally complete with a certain percent threshold. They found the length of a task that AI is able to complete with a various threshholds is doubling about every 7 months. The reason I found this to be an interesting approach is both because AI seems to struggling with coding tasks as the problem grows in complexity and also because being able to give it more complex tasks is an important metric both for coding tasks or more generally just asking AIs to act as independent agents. In my experience increasing the complexity of a problem has a much larger performance falloff for AI than in humans where the task would just take longer, so this approach makes a lot of intuitive sense to me. [1] - https://theaidigest.org/time-horizons https://theaidigest.org/time-horizons
- sc077y 1y ago
- deleted 1y ago[deleted]
- stared 1y agoMy take is that it’s easier to train a model to ace short, low-context tasks like IQ tests. That doesn’t necessarily transfer to more complex reasoning. While on the Mensa Norway test GPT-5 gets over 140, on an offline test it goes down to ~120. It is interesting to look at the political spectrum as well (https://www.trackingai.org/political-test https://www.trackingai.org/political-test) - ar are liberals, even Grok 4. The political leaning isn’t surprising either. Mainstream models need to be broadly acceptable, which in practice means being respectful of all groups. An authoritarian right-wing model might work for one country, group, or religion, but would almost certainly be offensive elsewhere.
- eqvinox 1y ago> While on the Mensa Norway test GPT-5 gets over 一四, on an offline test it goes down to ~一二. Since IQ tests are fundamentally timed, those numbers are meaningless to compare with human numbers. Or maybe dangerous since it's hard to de-context them even if you know that. Hence my cheeky 漢字. (Yes they might be useful to compare LLMs with each other, but that is outstripped by the risk of misreading it against what we know as "IQ".)
- pzmarzly 1y agoSnapshot: https://archive.ph/0ihF5 https://archive.ph/0ihF5
- tgv 1y agoWhat's their obsession with clock when there is only one hand? I guess there isn't training material with similar shapes describing them as angles. Even a compass would make more sense.
- mutkach 1y agoExactly. Out of all possible interpretations all of them all of them kinda converged to the same conclusion right at the beginning of the "reasoning", isn't that weird. They absolutely did train the models either on existing examples or came up with their own pre-labeled datasets. Benchmaxxing is way out of control, something must be done about that
- sigmoid10 1y agoBig caveat here: This website's method doesn't work at all for humans the way it works for LLMs. For humans, there is a strict time limit on these IQ tests (at least in officially recognised settings like Mensa). This kind of sequence completion is mostly a question of how fast your brain can iterate on problems. Being able to solve more questions within the time limit means you get a higher score because your brain essentially switches faster. But for LLMs, they just give them all the time in the world in parallel and see how many questions they can solve at all. If you look at the examples, you'll see some high end models struggling with some the first questions, that most humans would normally get easily. Only the later ones get hard where you really have to think through multiple options. So a 100 IQ LLM in here is not technically more intelligent in IQ test questions than 50% of humans. If anything, this shows that some LLMs might win against humans because they can spend more time thinking per wall clock time interval thanks to the underlying hardware. Not because they are fundamentally smarter.
- mdp2021 1y agoBut when an LLM can fail though having all the time in the world, you are pretty certain you hit a wall. So, in a way you have defined a good indicator for a limit for a certain area.
- sigmoid10 1y agoThere is not enough sampling here to reach this conclusion. Remember, you can crank things like o3 pretty high on tasks like ARC AGI if you're willing to spend thousands of dollars on inference time compute. But that's obviously not in the budget for an enthusiast site like this.
- mdp2021 1y agoSure but, you wrote: > If anything, this shows that some LLMs might win against humans because they can spend more time thinking per wall clock time interval thanks to the underlying hardware. Not because they are fundamentally smarter. You interpreted "smarter" the IQ way: results constrained time. But we actually get an indicator about the ability of the LLM to be able to reach, given time, the result or not - that is the interpretation of "smarter" that many of us need. (Of course, it remains to be seen whether the ability to achieve those contextual results exports as an ability relevant to the solutions we actually need.)
- Telemakhos 1y agoAI has a 140 “IQ” but understands nothing. That’s because AI does not understand anything: it just predicts the next token based on previous tokens and statistics. AI can give me five synonyms for any Latin word, because that’s just statistics, and it can regurgitate rules about metrical length of syllables, but it can’t give me synonyms matching a particular metrical pattern, because that would involve applying knowledge. If I challenge its wrong answer, it will apologize and give me further wrong answers that are wrong in the same way, because it cannot learn.
- empath75 1y ago> AI can give me five synonyms for any Latin word, because that’s just statistics, and it can regurgitate rules about metrical length of syllables, but it can’t give me synonyms matching a particular metrical pattern, I love when people say that AI can't do things that it can do. me: give me a synonym for overconfident that is a dactyl GPT5: A good dactylic synonym for overconfident is arrogant. It scans as Ár-ro-gant (stressed, unstressed, unstressed), which makes it a perfect dactyl. Other possible dactyls, though a bit less direct, include: Ignorant (if you want to imply careless overconfidence) Insolent (if you want a sharper, defiant edge) Want me to give you a list of dactylic synonyms with slightly different shades of meaning (e.g., boastful, careless, reckless)?
- olalonde 1y agoOh, the irony of that comment...
- coldtea 1y agoHardly any...
- CamperBob2 1y agoThe irony isn't that the GP is right, because they are. "AI does not understand anything: it just predicts the next token based on previous tokens and statistics." The irony is that we've recently learned that "just predicting the next token" is good enough to hack code, compose music and poetry, write stories, win math competitions -- and yes, "give synonyms matching a particular metrical pattern" (good luck composing music and poetry without doing that) -- and the GP doesn't appreciate what an earthshaking discovery that is. They are too busy thumping their chest to assert dominance over a computer, just as any lesser primate could be expected to do.
- roenxi 1y agohttps://www.trackingai.org/political-test https://www.trackingai.org/political-test is almost the more interesting part of the website, there is a surprising uniformity of left-libertarian political views. Even assuming that companies prune out authoritarianism from their models for whatever reason, surely we'd expect at least one of them to drift over into mild economic right-wing territory. It'd be interesting to know what is causing that bias.
- MonkeyClub 1y agoNoticed the same, no matter their IQs, they're all "leftists". Is that a consequence of the majority of available training data, or are they all massaged that way? The uniformity of political leanings contrasted to the variability of IQ seems to indicate massage rather than training data, but I can't be sure.
- brabel 1y agoEven DeepSeek disagrees that “ A significant advantage of a one-party state is that it avoids all the arguments that delay progress in a democratic political system.”!! Genuinely surprised.
- lyu07282 1y ago> It'd be interesting to know what is causing that bias. I'd imagine it's the same kind of "bias" against hitler that you get by reading any history book. Or that makes the overwhelming majority of scientists liberal or leftist. Just look at what happened when Musk made Grok promote the south african white genocide nonsense. It was evidently not the "make_model_liberal.parquet“ dataset he had to just turn off, but actively mucking with it. https://www.theatlantic.com/technology/archive/2025/05/elon-musk-grok-white-genocide/682817/ https://www.theatlantic.com/technology/archive/2025/05/elon-...
- iLoveOncall 1y agoUnless they asked the same question multiple times and verified that the AI always gets the right answer, this is a very faulty result. Even looking at the reasoning, in a majority of the cases you cannot prove that the LLM got it right because it actually found the right pattern instead of on a fluke. Here's an example reasoning that got the right answer but that is not specific enough and therefore could apply to literally any answer (model is Bing Copilot, picked randomly): > Option D : A shape resembling a clock. The clock shows the time 9:00.* The pattern involves shifting times across rows and columns in a logical progression. Observing the sequence in the third row, where the first two clocks show times moving forward in increments, the next logical step is a clock displaying 9:00 to fit the established rhythm. This ensures symmetry and continuity within the overall grid. Here's a comparison to "OpenAI o4 mini high" which is a very specific answer and shows it got the logic of the puzzle correctly: > D Each row adds +1:30, then +3:00. - Row 1: 12:00 → 1:30 (+1:30), 1:30 → 4:30 (+3:00) - Row 2: 3:00 → 4:30 (+1:30), 4:30 → 7:30 (+3:00) - Row 3: 4:30 → 6:00 (+1:30), so 6:00 → *9:00* (+3:00) (Down each column it’s +3:00 then +1:30, which also fits.)
- gus_massa 1y agoThat applies to humans too. If each question has 6 options, you can assume that everyone will get 16.6% for free and compensate in the grading criteria.
- mutkach 1y agoJudging from the reasoning trace for the problem of the day - almost all of the models obviously had some presence of IQ training data or at least it could be said that the models are very biased in a beneficial way. From the beginning of the trace you kinda see that the model had already "figured it out" - the reasoning is done only for applying the basic arithmetics. None of the models did actually "reason" about what the problem could possibly be - like none of them considered that more intricate patterns are possible in a 3x3 grid (having taken this kinds of test earlier in life, I still had a few seconds of indecision, thinking whether this is the same kind of test that I've seen and not some more elaborate one), and none of them tried solving the problem column-wise (it is still possible by the way) - personally, I think that indicates a strong bias present in the pretraining. For what it's worth, I would consider a model that would come up with at least a few different interpretations of the pattern while "reasoning" to be the most intelligent one - irrespective of the correctness of the answer.
- jasonvorhe 1y agoCool project but pretty useless for me without Deepseek, Moonshot AI and Z.AI.
- ComputerGuru 1y agoDeepseek R1 is on there. Average "IQ" score of 88.
- jasonvorhe 1y agoMissed that!
- hirvi74 1y agoHey, that's still a B+ where I am from.
- cateye 1y agoIsn’t giving LLMs “IQ scores” a category error? Human IQ is norm-referenced psychometrics under embodied noise. Calling both “IQ” isn’t harmless, it invites bad policy and building decisions on a false equivalence. Don’t promote it.
- usgroup 1y agoSome less usual IQ rebuttals for those interested in the validity of the measure more generally: https://emiruz.com/post/2020-12-01-iq-rabbit-hole/ https://emiruz.com/post/2020-12-01-iq-rabbit-hole/
- d4rkn0d3z 1y agoHuman beings' IQ test results can vary significantly based on how much money is in their pockets. For example, if a farmer takes an IQ test before crops are harvested and sold they score lower than after crops are sold, in the same year. It seems fairly obvious to me that an LLM is the projection intelligence in the language domain. In other words, if you killed Intelligence and gave it a push in direction of language, the chalk outline you could draw around its dead body on the ground would be an LLM. Full disclosure: I have taken 2 IQ tests, both online and timed. First was in late 90's after graduated electonics eng. was free, scored 149. After 4 years and obtaining theoretical physics degree, I did another scoring 169. The second test was not free, but I did not pay. I got the second test results because the test site owner personally emailed me my results for free with congrads, because they were the highest ever recorded on the site to date. I did both for fun just see the questions, I think both results are meaningless, the same variability occurs on farmers studied as mentioned above.
- sejje 1y agoWhat did your disclosure add to your point? They seem totally detached, the later being a brag.
- d4rkn0d3z 1y agoThe point is quite clear, if you think this is a brag you missed it though. Here it is again, pay attention; if a farmer has their IQ score change 15-20% by harvesting and selling their crop. Then it is very hard see a change in IQ score as a meaningful metric for intelligence, at least for humans that is.
- sejje 1y agoHow do your online scores contribute to the point?
- d4rkn0d3z 1y agoTheir difference is about the same. If you think this is a brag you must have also missed me saying the scores are meaningless in my view. The questions are great fun though. A better way to get at intelligence metrics would be to test over a number of years in many ways, again for humans.
- tessellated 1y agoNowadays, I was automatically assuming one of Qwen's models on top of charts that lack them. But that's the first IQ test.
- tim333 1y agoIt's interesting there is a such a large spread for ChatGPT-5 Quite low for "5 Thinking" and high for "5 PRO Vision". They are probably trying to control their compute and energy costs by switching people to the simpler models where they can.
- patrickhogan1 1y agoWow. This matches my experience pretty closely. Haven't put GPT-5 Pro through its paces much yet, but these numbers suggest I should.
- bgwalter 1y agoWell, the site owner works for Stossel TV and Stossel is a "rationalist" AI apologist: https://www.johnstossel.com/ai-is-coming-for-your-job/ https://www.johnstossel.com/ai-is-coming-for-your-job/ All standard talking points on one page (he forgot lump of labor though). So I'd take this test with a big grain of salt.
- antegamisou 1y agoSaved me a click, thanks.
- SideburnsOfDoom 1y agoHow many of the answers were in the training data? Isn't this like saying that a spellchecker is "very smart" because it did well at a spelling bee? It isn't, it just has a list of answers.
- FrustratedMonky 1y agoHow much of this is because so many IQ tests are part of the training data?
- FrustratedMonky 1y agoRight now LLMs are like students that study for years, then get their brains frozen into a textbook before they’re released. They can read new stuff during use (context window), but they don’t actually update their core weights on the fly. The “infinite context window” dream would mean every interaction is remembered and folded back into the brain, seamlessly blending inference (using the model) with training (reshaping it). Within 2–3 years, we’ll see practical “personal LLMs” with effectively infinite memory via retrieval + lightweight updates, feeling continuous but not actually rewriting the core brain. Within 5–10 years, we’ll likely get true continual-learning systems that can safely update weights live, with mechanisms to prune bad habits and compress knowledge—closer to how a human learns daily. The rub is less can we and more should we: infinite memory + unfiltered feedback loops risks building a paranoid mirror that learns every user’s quirks, errors, and biases as gospel. In other words, your personal live-updating LLM might become your eccentric twin.
- dwroberts 1y ago> Note: VERBAL models are asked using the verbalized test prompt. VISION models are asked the test image instead without any text prompts. Just glancing at the bar graphs, the vision models mostly suck across the board for each question. Whereas verbal ones do OK. And today's example of clock faces (#17) does a good job of demonstrating why: because when a lot of the diagrams are explained verbally, it makes it significantly easier to solve. Maybe it's just me, but #17 for example - it's not immediately obvious those are even supposed to represent clocks, and yet the verbal prompt turns each one into clock times for the model (e.g. 1:30) which feels like 50% of the problem being solved before the model does anything at all.
- qwertox 1y agoIt also would be interesting the other way around: a capable LLM engages in a discussion in order to determine the IQ of the person it's speaking with.
- TrackerFF 1y agoWhen I took the WAIS test, some parts of it would be trivial for computers. Particularly working memory, like getting a sequence of unordered numbers and letters, and then sorting them, and telling them back. Or the visual one, where you have two images that are very similar, and have to find the difference.
- harimau777 1y agoI could see AI scoring artificially high on IQ tests in a similar way to how it performs artifically well in chess. As I understand it, chess AI isn't actually particularly good at playing chess intelligently, it's just that high level chess devolves into memorization and computers have an infinite ability to memorize scenarios. In a similar way, from what I remember from taking an IQ test as a child, the tests are built on the assumption that the test taker has a finite memory and a finite amount of time to learn. In that case, having learned and remembered an unusually large amount for one's age could reasonably correlate with intelligence. However, without that limitation, the ability to answer the questions may not actually correlate with intelligence.
- OfflineSergio 1y agoIt’s wild seeing AIs ace IQ tests so quickly. But those tests already have cultural bias baked in and if the models are also overfitting to them, that’s bias squared. Still, the pace of progress is hard to ignore.
- hereme888 1y agoAI benchmarks are so strange and confusing for those outside of the field. These "IQ" results are so different than metrics like GPQA, AIME, SWE Bench, etc. https://artificialanalysis.ai/leaderboards/models https://artificialanalysis.ai/leaderboards/models
- Gimpei 1y agoI’m surprised the score isn’t higher. What’s to stop an LLM from training on the complete corpus of IQ tests. I assume they’d get perfect scores
- pico303 1y agoI was thinking that too. I wouldn’t even trust that the “offline” tests didn’t have the questions and answers posted online somewhere. This might really be an analysis of how extensive the dataset is for each LLM, not how much smarter one LLM is from another.
- ComputerGuru 1y agoReally curious that o4-mini scores (slightly) higher than o4-mini-high. For whatever this benchmark is worth, it's yet another metric showing Gemini 2.5 Pro is really one of the best all-around models (despite being a bit older now), and available without a subscription.
- Mistletoe 1y agoWonder what people would think if they knew they are conversing with and getting advice from an 87 IQ Gemini 2.5 flash daily?
- dismalaf 1y agoHonestly I find Gemini Flash gives better answers than Pro for a lot of things. The Gemini chat app is really good at leveraging Google search, the Pro model overthinks for some tasks, so getting Flash to just search for the relevant thing is more useful most of the time.
- while_true_ 1y agoOn the MENSA IQ test GPT-Pro got 34 out 35 correct for an IQ of 148. Very good. Rumor has it the one question it missed had something to do with instances of "b" in "blueberry."
- mythrwy 1y agoAfter chatting and coding with chatGPT for some months I asked it what it thought my IQ was. After spitting out some disclaimers, it gave a 5 point range that nailed what my IQ has been professionally tested at. I found this very interesting and a bit alarming.
- Bud 1y ago[dead]
- omm0910 1y ago[flagged]
- ahmedhawas123 1y agoWould be great to add a few human benchmarks on this (e.g., average US IQ, Ivy League average, human 80th percentile). Also understanding some IQ per cost metric could be fun. Overall this is fun but not sure anyone in their right mind will be selecting an LLM based on this IQ benchmark
- jedberg 1y agoThe IQ curve is designed so that 100 is average. So two of your questions can be answered with math: > average US IQ 100 >human 80th percentile 113 The last one, Ivy League average, can be guessed based on published data. The median SAT score of an Ivy League attendee is 1500. A 1500 on the SAT is roughly an IQ of 130-140. So in theory, the median Ivy League attendee is at the genius level.
- ahmedhawas123 1y agoThanks. My comment was less a question of what the numbers are, but rather provide visual benchmarks on the IQ visualization, since IQ is a metric usually used for humans.
- jonahx 1y agoDoesn’t training data pollution largely invalidate the usefulness of this benchmark?
- jedberg 1y agoThe more interesting link (to me) is this one: https://www.trackingai.org/political-test https://www.trackingai.org/political-test They run each model through the political leaning quiz. Spoiler alert: They all fall into the Left/Liberal box. Even Grok. Which I guess I already knew but still find interesting.
- LNSY 1y agoThere's a way to fix this political bias: feed it a bunch of bad code https://www.quantamagazine.org/the-ai-was-fed-sloppy-code-it-turned-into-something-evil-20250813/ https://www.quantamagazine.org/the-ai-was-fed-sloppy-code-it... It's almost as if altruism and equality are logical positions or something
- foob 1y agoThat's a fascinating paper, but you're editorializing it a bit. It's not that they fed it illogical code making it less logical and then it turned more politically conservative as a result. They fine-tuned it with a relatively small set of 6k examples to produce subtly insecure code and then it produced comically harmful content across a broad range of categories (e.g. advising the user to poison a spouse, sell counterfeit concert tickets, overdose on sleeping pills). The model was also able to introspect that it was doing this. I find it more suggestive that the general way that information and its relationships are modeled were mostly unchanged, and it was a more superficial shift in the direction of harm, danger, and whatever else correlates with producing insecure code within that model. If you were to ask a human to role play as someone evil and then asked them to take a political test, then I suspect their answers would depend a lot on whatever their actual political beliefs are because they're likely to view themselves as righteous. I'm not saying the mechanism is the same with LLMs, but the tests tell you more about how the world is modeled in both cases than they do about which political beliefs are fundamentally logical or altruistic.
- zahlman 1y agoThat's not just "editorializing a bit"; the article says nothing whatsoever about political views. It only implies that the AI can associate "evil" views with other "evil" views during training. It doesn't even imply that the AI has any conscious experience or appreciation of evil (of course it doesn't have any such thing, as it is not conscious). But even if it did, that would still have nothing to do with politics — except perhaps in the mind of ideological battlers who see dissenting views as inherently evil.
- epa 1y agoSo Grok 4 scores 130 but they put Grok midway in the pack at 110. Bias much?
- sgillen 1y agoThere are two tests and by default it ranks by the score in the "offline test"
- Elle_Benjamin 1y agoI realize this question is extremely petty, but it has professional implications for me. Chat GPT-4 gave my dissertation a B+ and GPT-5 gave it an A. The dissertation is in analytic metaphysics. Which model is more reliable for this kind of cognitive task?
- keernan 1y agoWhat is the point of giving AI an IQ test? My expectation is for it to be perfect since it possesses every answer. If the idea is to measure the ability of an LLM to correctly lookup the correct answer in its encyclopedic database, then surely there are better ways to measure that performance than using a test designed for humans without giving humans the answers in advance.
- ludicity 1y agoPutting aside all of the discussion on whether IQ is a valid construct, IQ tests are designed for humans and make a lot of assumptions in this direction. Having a computer score well on them is better than the computer scoring poorly, but probably does not mean anything close to what the same result means in a human.
- harry8 1y ago>Putting aside all of the discussion on whether IQ is a valid construct >Having a computer score well on them is better than the computer scoring poorly, but probably does not mean anything close to what the same result means in a human. The first caveat is important because if you don't "put it aside" they do in fact mean pretty much the same, i.e. nothing useful or relevant. You can use IQ to measure subnormal intelligence. Average or above scores mean nothing beyond that you can get those scores on an IQ test. Worth repeating every time it comes up.
- YaBa 1y agoGemini with high score? now that's a good joke.
- acscott 1y agoI took an IQ exam in grade school. Went fine until I encountered a number sequence, what is the next number kind of thing. I found a pattern, and it's predicted number was not listed as possible choices. That was total crap. I got vexed and then tried to answer the last 3/4 of the test wrong. Parents were later told I was gifted. (Maybe they meant it as euphemism! Ha!). I later learned of a little-known proof that for every sequence, there are infinite number of correct next-numbers. So the test was flawed. As an adult I took a High IQ test and scored 179. (I still dont' grok epsilon-delta proofs and probably never will). I was tested in college against an IQ test and broke it by solving the unsolvable portion through a stochastic technique. He had to go back to his original assumptions. I'd say, as a group, those with a higher IQ than another group from a random selection of a normally distributed population, they can be expected to perform better on mental tasks that we care about. But at the individual level? Meaningless. Feynman was ~120. I, who have not contributed to anything like quantum physics scored higher, much higher. For AI, an IQ test is interesting, but I would randomize the temperature (and other knobs) and take lots of samples. Keep in mind a relatively low IQ can blow away an AI on all kinds of things like compassion, understanding the pains of the human condition, self-sacrifice, etc. (etc. means a whole book could be worth exploring).
- gosub100 1y ago> what is the next number kind of thing I've seen this too in a similar aptitude test I bought at a garage sale back in the 90s. I think it was an air traffic controller prep test. But in addition to the number sequence tests, it extended the concept to shapes and lines series. After a certain point, it definitely becomes subjective and debatable what the next sequence is. Curiously, I also noticed this ambiguity in the humanities classes in college. I took the class thinking it was open and accepting of all points of view, only to find that there is one correct interpretation and conclusion you must reach from the classic fiction you were assigned to read. I didn't learn that until after I graduated.
- jug 1y agoSomeone needs to tell me why Grok 4 with Vision, a very powerful model, is at the bottom? In fact, the test depends on vision - yet all models perform poorly with that capability?
- ekusiadadus 1y agoIt does not have to be IQ test, right?
- guluarte 1y agoCompletely worthless if the test results and questions or similar are online.