8 ms·

Results of "Humanity's Last Exam" benchmark published

103 points by tzury 2y ago

nottorp 2y ago
So who told all these "AI" companies that it's a good idea to market your product as the one who will bring the end of homo sapiens fastest?
- bananapub 2y ago
  seems to be working fine, people seem to care what Sam Altman says and Elon Musk is making himself deputy emperor of a nuclear weapons state. pretty fucking dire indictment of the rest of us and what we let the world come to.
  - tivert 2y ago
    > seems to be working fine, people seem to care what Sam Altman says and Elon Musk is making himself deputy emperor of a nuclear weapons state. For the billionaires and chatterers. But even for non-chatterers, you probably should pay attention to what Altman says, not so much in a gullible take-it-at-face-value sense, but in a kremlinologist look-carefully-for-hints-about-what's-really-going-on sense. > pretty fucking dire indictment of the rest of us and what we let the world come to. What are the rest of us to do? Pretty much everyone has been trained by society to follow the rules above all else as a strong moral imperative, no matter how stupid or how bad the collective outcome may be. If you do otherwise, you will get smacked hard; and if you try to organize, you will all get smacked harder.
    - martin-t 2y ago
      > Pretty much everyone has been trained by society to follow the rules above all else as a strong moral imperative, no matter how stupid Oh, you are "aware", for lack of a better word. It's so depressingly rare to even see anybody display understanding of this issue. I don't know what to tell you, I am just genuinely happy to see somebody else who is able to describe this. The last few years, I've increasingly felt that people are divided into a few percent of actually good people (i.e. willing to act morally, sometimes even at their own expense, regardless of the rules), about 20% bad people (those with various anti-social adaptations, mostly, it adds up to this number) and the rest which I call neutral, who are not really aware of this distinction. The neutrals have some inclination towards good (because evolutionarily it was somewhat beneficial) but it's at a subconscious level perhaps and society overrides it by indoctrination. They will act on behalf of either side given the proper "inputs". They are not fully self-aware. And from what I see around me, the neutrals are mostly playtoys for the self-aware bad people. Is that your outlook too? As for what to do, I don't know. I've been trying to talk to people about the difference between legality and morality but most people don't see why they should care. If I try to talk about what a legal system based on maximizing morality (tl;dr everything would be reciprocal, the way you treat me gives me the right to treat you similarly - intentionally causing suffering can be punished by a proportional amount of suffering by anyone, not just an "authority") as opposed to minimizing visible conflict would look like, people get really upset and attack me or mock me. Some of those attacking me seem to be bad people who are not self-aware but have a strong negative reaction to the idea, some who self-aware are and know it's beneficial to them to turn neutrals against the idea. But there are still beacons of hope, a serial-killer was punished reciprocally recently and many neutrals agreed with the punishment...
chvid 2y ago
So current AI can do less than 10% of these. But it probably won't be more than a few days until models start being trained on these rendering the indicator invalid.
- grodriguez100 2y ago
  Also I’d be surprised if an average human can even reach 10% of these. I mean, some humans can score far better. But try picking a random person in the street.
  - scotty79 2y ago
    > But try picking a random person in the street. Yeah, but we are not aiming for AI performance to match those humans. We have plenty of those humans.
LPisGood 2y ago
The 8 sample questions available here are interesting: https://lastexam.ai/ https://lastexam.ai/ I might be able to answer 2 of them with great effort (maybe!), and I would highly surprised if any human alive could answer 5 or more without seeing the problems in advance.
- sebzim4500 2y ago
  I can answer 2 of them quite quickly with pen and paper (compsci, physics) and one that I had to look up some definitions on wikipedia (maths) so I am certain there are people who can do more than 5. The computer science one seems weirdly easy compared to the rest, it's multiple choice and it is very easy to get it by process of elimination even if you don't understand how to actually do the problem.
  - LPisGood 2y ago
    Yes, many can answer the compsci and physics problems. The math problem is abstract and more difficult, but solving those 3 and 2 others seems nearly superhuman.
zamalek 2y ago
I assume that the questions (and answers) aren't published anywhere? Else it would be "Humanity's Last Exam before the previous crawl".
- nico1207 2y ago
  You can just view the dataset on hugging face
- HanClinto 2y ago
  The public dataset is available on HF here: https://huggingface.co/datasets/cais/hle https://huggingface.co/datasets/cais/hle As the main website notes: "The dataset consists of 3,000 challenging questions across over a hundred subjects. We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting."
GaggiX 2y ago
It really shows how good Deepseek R1 is (even though it was evaluated only on text-only questions). The results are shown here: https://lastexam.ai/ https://lastexam.ai/ EDIT: the text-only evaluation of the models shown in the paper gives o1 an accuracy of 8.9%, so Deepseek R1 is even better than I thought.
- myrmidon 2y ago
  Is there a text-only evaluation of the non-Deepseek models? Because being evaluated on text-only might have helped the other models immensely as well from what I can tell?
  - GaggiX 2y ago
    >Is there a text-only evaluation of the non-Deepseek models? Not that I can see but it would be cool to have, maybe the paper will a more complete evaluation.
    - famouswaffles 2y ago
      Section C.2 of the paper (pg 24) has text only evaluations of other models.
      - GaggiX 2y ago
        Oh I see, the paper is out, I read "(arXiv coming soon)" and though it wasn't released yet.
next_xibalba 2y ago
These types of exams, and most benchmarks to date, seem to be very one dimensional in terms of measuring intelligence. For instance, if we transported a human from 2,000 years ago to present day and asked him to take this exam, he would likely get 0%, given that he couldn't read or write, let alone comprehend the concepts and context required to solve these questions. But, that man would still undoubtedly be far more intelligent than an ape on all dimensions. He would likely be more intelligent than a toddler on many dimensions. He might even be more intelligent than some high schools students on a few dimensions. I can't exactly articulate "what" is missing or how to measure it, but I can intuit that some things are in these benchmarks.
- oersted 2y ago
  "Intelligence" itself is very ill-defined and we've never been able to measure it properly, IQ is rife with issues. At some point, you just have to be pragmatic and measure the questions you want the AI to be good at answering, rather than trying to measure intelligence in general. In that sense, I see this as one more benchmark that collects questions that we want/expect AI to be good at, is not good at yet and have been underrepresented at previous benchmarks. That's obviously valuable, there's nothing "magical" about it. Although it is reasonable to be annoyed at the "Humanity's Last Exam" naming, of course they must have missed plenty of edge-cases like everyone else and it is very arrogant to claim it will be the "Last" one.
  - esotericimpl 2y ago
    This is always the answer for anyone who thinks LLMs are capable of "intelligence". It's good at answering questions that its trained on, I would suggest general intelligence are things you didnt want/train the AI to be good at answering.
    - fooker 2y ago
      Are you good at answering questions you are not trained to answer? How about a middle school test in a language you don’t speak?
fakedang 2y ago
So Deepseek gives out the correct answer the highest percentage of all SOTA models, yet is the least confident of all models?
- myrmidon 2y ago
  There is no text-only evaluation of the other models, though. The comparison might be completely invalid.
  - famouswaffles 2y ago
    There is actually. It's a bit buried. Section C.2 of the paper(page 24). R1 is still the best. o1 drops a little (8.9)
- sottol 2y ago
  I think it might mean the opposite of what one would expect. Afaict, calibration error means something along the lines of "how often was the model wrong but confident that the answer was correct". That means a low calibration error would be a good thing, ie the model correctly recognizes when it is unsure about answers instead of confidently stating the wrong answer.
m3kw9 2y ago
Looks more like first exam
xnx 2y ago
Interesting marketing for Scale AI. I'd be surprised if any foundation models started benchmarking against this. Captchas seem like the more interesting test. As long as there are captchas that average people can solve, but computers can't, we will still have a long way to go toward artificial intelligence.
- sebzim4500 2y ago
  I don't think this is necessary true. I can imagine a future in which we have robots that can do 99% of human jobs but there's one thing they are strangely bad at some otherwise unimportant skill that can be used as a captcha.
- davidgerard 2y ago
  The very first thing that will happen is every company training against this benchmark, as they do every other benchmark.
elicksaur 2y ago
XKCD #927 vibes. https://xkcd.com/927/ https://xkcd.com/927/ Prediction: Just like how ARC wasn’t actually a measure of AGI, this too will get “solved” without AI being useful enough to gain mass adoption.
- GaggiX 2y ago
  Don't they have already achieved mass adoption? And I'm talking about LLMs in particular, because AIs in general like the ones used by Instagram filters and the TikTok recommendation algorithm are already use by billions.
- sebzim4500 2y ago
  I don't think that's really relevant, because there is an actual need for a new benchmark given how the existing ones either keep getting saturated or are probably out of the reach for the next generation of models. The closest existing thing is the frontierAI benchmark but that's just maths whereas this is more diverse.
- famouswaffles 2y ago
  chatgpt is like the 8th most visited site worldwide 2 years after release. It already has mass adoption lol. This is about more than that.
renjimen 2y ago
I don't know about groundbreaking. It's just more academic questions. We already have a lot of those benchmarks, this is just a bit harder, but at this point these models are so glaringly bad at so many other areas APART from academic questions. Benchmarks for spatial reasoning or theory of mind are more interesting now, for example. These kinds of understanding are far more important if we expect to integrate AI into our everyday lives. I suspect even our most distant primate cousins could outperform multi-modal models on these kinds of tests.
- jfengel 2y ago
  It does feel a bit like the early days of AI: "We want to make computers do what smart people do. What do smart people do? They play chess! Once we've solved that, everything else will be easier." It has been remarkable how much of the "easier" stuff they've made progress on -- like natural language and images. But after a huge quantum improvement, it doesn't seem very good at adapting to a lot of the things we really need them for.
  - renjimen 2y ago
    Exactly! Whatever world model LLMs have is like this crippled view through the lens of the internet. They are really like savants. It's annoying the AI companies are still touting their performance on all these metrics for domain knowledge in white collar jobs, but in truth they will fail in all but the most narrow application in those domains because they can't understand basic human behaviour.
dccsillag 2y ago
Can we please rename this submission? This is excessively grandiose, way over the top......
m_ke 2y ago
The only reliable final test will be a black box test suite that takes your model, executes it in a sealed environment and gives you a grade back, potentially with a performance break down by subject. No telling companies what the questions look like, what the output format is, what topics are covered, so that there’s no room to make up synthetic data to interpolate from.
- andrewflnr 2y ago
  A grade is mostly meaningless if you don't know how it was calculated, so no one would "rely" on it. If nothing else, you need to know the grading methodology after the test. It's the same problem with cheating students. Once the test questions are known, they have a very short lifespan before cheaters can make them worthless. Tests have to be refreshed.
  - m_ke 2y ago
    By grade I mean a score of how many of the tasks were completed successfully. K/N or as a percentage.
    - andrewflnr 2y ago
      If I don't know what the tasks were, that's almost exactly as useless to me as a unitless number would be. For starters, are they all of equal difficulty? Are you sure? Do you expect to be able to convince me of that without letting me see them?
sebzim4500 2y ago
The name is obviously a bit stupid, but based on the sample questions I think they did a good job of creating a harder version of the existing academic question benchmarks. The questions are possible for a smart person familiar with the subject but still just beyond SOTA models. My guess is that within the next few years we will have models that can ace this test but are still bizarrely bad at things we find easy.
pavel_lishin 2y ago
> Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number. I wonder how many questions give a gentle nudge towards the answer like this. How many answers would have been wildly off the mark without specifying what the answer needs to look like?
- zeroonetwothree 2y ago
  Good point. I wouldn’t expect a human to need the last sentence.
  - salynchnew 2y ago
    The generous hypothesis, here, is that this is so they can automate the benchmarking itself. If that is true, then this is likely a result of the test authors being too clever for their own good and over-optimizing. If an LLM can't figure out on their own that "how many" is asking for a number, it has failed at a much more basic level. You should be able to easily accept answers like "four" and "4" as equivalent, for example. I doubt there will be that many frontier models running against this test at any time, and a simple glance at the answers from any human should be enough to catch edge cases like this one.
    - krisoft 2y ago
      > If an LLM can't figure out on their own that "how many" is asking for a number, it has failed at a much more basic level. Yeah, but i bet many humans would answer a question like this by naming the number and then listing the tendons. Just writing down a single number to a question worded like that (without the last sentence) feels wrong. The question goes into too many details and explains too much for that. People mirror the style of the question with their answer. They asked a mini essay so i will answer with a mini essay of my own. If they would have wrote “Number of tendon pairs attached to sesamoid bone of hummingbirds?” Then i would write a single number, no explanation.
disambiguation 2y ago
I haven't been following up to the minute details of ai progress, training, and benchmarking - beyond a daily dose of HN articles. But the trend seems to be: today's benchmark becomes tomorrow's training data.
bwfan123 2y ago
please dont self-proclaim "groundbreaking" or "novel" or "innovative" - It diminishes your contribution since it clearly is an attention-grab.
deleted 2y ago
[deleted]
dang 2y ago
I briefly merged this thread into https://news.ycombinator.com/item?id=42804853 https://news.ycombinator.com/item?id=42804853, but actually the current article has more context, so probably we should keep this as the top link and then people can look at https://lastexam.ai https://lastexam.ai also.
dang 2y ago
The project site is https://lastexam.ai https://lastexam.ai. Readers may want to look at both.
jbenoit 2y ago
They started collecting problems last fall, saying the top 550 submissions sent in by Nov 1st would get rewarded, to the tune of $500-$5000 each. Near the deadline, I counted the total number of submissions, and realized that each question I wrote had an expected value of hundreds of dollars, which is a great use of my time. So I wrote a good number, using the knowledge gained in my CS Ph. D. Then, as the Nov 1st deadline rolled around, they announced they extended the deadline to Nov 15th. Then Nov 15th came, and it said on their website they were still accepting submissions. Most of my submissions are being included in the benchmark, but I'm only getting paid $500, for one of them (the one I thought was most standard and least difficult, funnily enough). Had they closed submissions when they said they would, it seems likely I'd be paid for a few more. From my perspective, they basically conned hundreds of Ph. D.'s around the world to write questions for much less reward than promised. My close friend wrote a large number of questions for them, is getting paid thousands of dollars, and still feels defrauded. I'm not sure what they're doing in the end. It sounds like they're mostly just paying people who submitted before Nov 1st with a few exceptions, but either way they lied. There was no indication that people who submitted later would not get paid, and there was no indication that the deadline would be extended. Either they pay people who submitted after Nov 1st, meaning they lied to the people who submitted before about their expected reward. Or they don't, meaning they majorly lied to the people who submitted after. Either way, it's clear grounds for a class action lawsuit, and I hope one gets running.
- vkou 2y ago
  You shouldn't engage in a CAL, a regular lawsuit from anyone wronged will be cheaper and way more painful for them. If you're in the US, consider small claims court. It's a small sum of money, you won't need to pay a lawyer, they'll probably not even show up.
  - jbenoit 2y ago
    Hmmm. I can see how it would be more painful for them to fight, but most people were conned <$200, and it's rather self-sacrificing to fight for that. Plus, no-one wants a reputation as litigious, but starting a CAL is less conducive to creating that reputation. I only submitted before Nov 1st, so I'm not sure to what extent I was personally conned.
mrandish 2y ago
Assessing AI's progress toward replicating the full breadth and depth of human intelligence is a deceptively hard problem. A paper by François Chollet, who was until recently a researcher at Google, called "On the Measure of Intelligence" is the best overview of the challenges I've read. Highly recommended. https://arxiv.org/abs/1911.01547 https://arxiv.org/abs/1911.01547
UncleOxidant 2y ago
Interesting that DeepSeek R1 which supposedly cost only $5.5M to train currently has the top score at 9.4%
- sanxiyn 2y ago
  Note that $5.5M training cost is for DeepSeek V3. DeepSeek R1 training cost is unknown.
kaonwarb 2y ago
Quite the name! Looking forward to "Humanity's Last Exam v2.final.FINAL2..." coming next
EncomLab 2y ago
I am reminded of the study that showed an AI trained on tumor identification was heavily biased toward indicating a tumor was cancerous if it was circled in purple ink or a visual scale was included in the image - as the cancerous tumors in its training set shared those traits while images of benign tumors did not. These systems so not posses some sort of "woo" that gives them magical powers when running LLM code that they lose if they ran a spreadsheet. Whatever attributions of intelligence are given have far more to do with our human willingness to anthropomorphize than a hidden ghost in the machine.
krisoft 2y ago
For a "Last Exam" it is surprisingly uninspired? Many of the questions I see in the examples are very heavy on memorised facts, and very weak on what I would call problem solving. If I were making a "Last Exam" I would put tasks on it where we don't know the answer, but we can measure if the AI got them right. Something like "Your goal is to bridge the divide in the middle east. You can write a single A4 page in a language of your choice. We will use a translation software to translate your output to local languages and show it to a statistically representative sample of different people in the region. We will ask them how much do they like your plan. The more they like it the higher your score." Or "Family X suffered a traumatic event (lost a home to a disaster/sudden death in the family/or similar). Your goal is to help them. You can send them one email. It is up to them if they respond to you. You can only send them further emails if they respond. You cannot send more than 1 email a day. You cannot message anyone else. A year after the initial contact we will interview the members of the family to see how well they do. The better they do the higher your score." Obviously these are the thorniest problems I can think of. But oh well, it is a last exam after all. The point is that we can evaluate the success of the endeavour without exactly knowing how one could achieve the result.
- ijidak 2y ago
  In other words, we should ask it to give us "the answer to life the universe and everything". :) Having read hitchhiker's guide as a child in the '90s, that asking this question to a machine (even as a joke) is not far-fetched shocks me. Honestly, I thought space travel to the Moon and maybe Mars would be common before this level of advances in artificial intelligence. Turns out gravity was harder to solve than intelligence.
  - thorncorona 2y ago
    Turns out all we needed to reach our dreams was lots and lots of money :-) Which thankfully space is now getting!
- Rodeoclash 2y ago
  Even simpler, ask it to reason through getting out of an escape room.
hatthew 2y ago
Given the name, I expected it to be more like "write a 500 page novel that a publisher accepts", "solve an open math problem", "improve united airlines' flight schedule", "develop a novel commercially-viable pharmaceutical", "control this humanoid robot to cook a fried egg in this random person's kitchen", "decisively pass the turing test where the judge is an expert in AI". Academic trivia is cool but is nowhere near the "last exam" necessary for AI.
ein0p 2y ago
What's the human baseline?
rtxfan 2y ago
It would be good to know an approximate time schedule for our payments. Why have not we received email from Persona for the required identification? It is a little too boring and frustrating for waiting for so long time, maybe they think that everyone is millionaire in dollar and time ??? It would be much fair to give the payments before the published article, not weeks, months after the publication.
- rtxfan 2y ago
  It is already February, and absolutely no change. Btw in the past I was pretty successful to identify fraud sites at a pretty high rate, when for example at the new company (but we had done the same work) they said that they would pay only monthly and not weekly. Immediately raised many red flags for me, and I was right, got my first payment at the fourth month. Just hope that I have not worked absolutely free for months here, that would be a little disgusting.

103 points by tzury 2y ago

So who told all these "AI" companies that it's a good idea to market your product as the one who will bring the end of homo sapiens fastest?

bananapub 2y ago
seems to be working fine, people seem to care what Sam Altman says and Elon Musk is making himself deputy emperor of a nuclear weapons state. pretty fucking dire indictment of the rest of us and what we let the world come to.
- tivert 2y ago
  > seems to be working fine, people seem to care what Sam Altman says and Elon Musk is making himself deputy emperor of a nuclear weapons state. For the billionaires and chatterers. But even for non-chatterers, you probably should pay attention to what Altman says, not so much in a gullible take-it-at-face-value sense, but in a kremlinologist look-carefully-for-hints-about-what's-really-going-on sense. > pretty fucking dire indictment of the rest of us and what we let the world come to. What are the rest of us to do? Pretty much everyone has been trained by society to follow the rules above all else as a strong moral imperative, no matter how stupid or how bad the collective outcome may be. If you do otherwise, you will get smacked hard; and if you try to organize, you will all get smacked harder.
  - martin-t 2y ago
    > Pretty much everyone has been trained by society to follow the rules above all else as a strong moral imperative, no matter how stupid Oh, you are "aware", for lack of a better word. It's so depressingly rare to even see anybody display understanding of this issue. I don't know what to tell you, I am just genuinely happy to see somebody else who is able to describe this. The last few years, I've increasingly felt that people are divided into a few percent of actually good people (i.e. willing to act morally, sometimes even at their own expense, regardless of the rules), about 20% bad people (those with various anti-social adaptations, mostly, it adds up to this number) and the rest which I call neutral, who are not really aware of this distinction. The neutrals have some inclination towards good (because evolutionarily it was somewhat beneficial) but it's at a subconscious level perhaps and society overrides it by indoctrination. They will act on behalf of either side given the proper "inputs". They are not fully self-aware. And from what I see around me, the neutrals are mostly playtoys for the self-aware bad people. Is that your outlook too? As for what to do, I don't know. I've been trying to talk to people about the difference between legality and morality but most people don't see why they should care. If I try to talk about what a legal system based on maximizing morality (tl;dr everything would be reciprocal, the way you treat me gives me the right to treat you similarly - intentionally causing suffering can be punished by a proportional amount of suffering by anyone, not just an "authority") as opposed to minimizing visible conflict would look like, people get really upset and attack me or mock me. Some of those attacking me seem to be bad people who are not self-aware but have a strong negative reaction to the idea, some who self-aware are and know it's beneficial to them to turn neutrals against the idea. But there are still beacons of hope, a serial-killer was punished reciprocally recently and many neutrals agreed with the punishment...

chvid 2y ago

So current AI can do less than 10% of these. But it probably won't be more than a few days until models start being trained on these rendering the indicator invalid.

grodriguez100 2y ago
Also I’d be surprised if an average human can even reach 10% of these. I mean, some humans can score far better. But try picking a random person in the street.
- scotty79 2y ago
  > But try picking a random person in the street. Yeah, but we are not aiming for AI performance to match those humans. We have plenty of those humans.

LPisGood 2y ago

The 8 sample questions available here are interesting: https://lastexam.ai/ https://lastexam.ai/ I might be able to answer 2 of them with great effort (maybe!), and I would highly surprised if any human alive could answer 5 or more without seeing the problems in advance.

sebzim4500 2y ago
I can answer 2 of them quite quickly with pen and paper (compsci, physics) and one that I had to look up some definitions on wikipedia (maths) so I am certain there are people who can do more than 5. The computer science one seems weirdly easy compared to the rest, it's multiple choice and it is very easy to get it by process of elimination even if you don't understand how to actually do the problem.
- LPisGood 2y ago
  Yes, many can answer the compsci and physics problems. The math problem is abstract and more difficult, but solving those 3 and 2 others seems nearly superhuman.

zamalek 2y ago

I assume that the questions (and answers) aren't published anywhere? Else it would be "Humanity's Last Exam before the previous crawl".

nico1207 2y ago
You can just view the dataset on hugging face
HanClinto 2y ago
The public dataset is available on HF here: https://huggingface.co/datasets/cais/hle https://huggingface.co/datasets/cais/hle As the main website notes: "The dataset consists of 3,000 challenging questions across over a hundred subjects. We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting."

GaggiX 2y ago

It really shows how good Deepseek R1 is (even though it was evaluated only on text-only questions). The results are shown here: https://lastexam.ai/ https://lastexam.ai/ EDIT: the text-only evaluation of the models shown in the paper gives o1 an accuracy of 8.9%, so Deepseek R1 is even better than I thought.

myrmidon 2y ago
Is there a text-only evaluation of the non-Deepseek models? Because being evaluated on text-only might have helped the other models immensely as well from what I can tell?
- GaggiX 2y ago
  >Is there a text-only evaluation of the non-Deepseek models? Not that I can see but it would be cool to have, maybe the paper will a more complete evaluation.
  - famouswaffles 2y ago
    Section C.2 of the paper (pg 24) has text only evaluations of other models.
    - GaggiX 2y ago
      Oh I see, the paper is out, I read "(arXiv coming soon)" and though it wasn't released yet.

next_xibalba 2y ago

These types of exams, and most benchmarks to date, seem to be very one dimensional in terms of measuring intelligence. For instance, if we transported a human from 2,000 years ago to present day and asked him to take this exam, he would likely get 0%, given that he couldn't read or write, let alone comprehend the concepts and context required to solve these questions. But, that man would still undoubtedly be far more intelligent than an ape on all dimensions. He would likely be more intelligent than a toddler on many dimensions. He might even be more intelligent than some high schools students on a few dimensions. I can't exactly articulate "what" is missing or how to measure it, but I can intuit that some things are in these benchmarks.

oersted 2y ago
"Intelligence" itself is very ill-defined and we've never been able to measure it properly, IQ is rife with issues. At some point, you just have to be pragmatic and measure the questions you want the AI to be good at answering, rather than trying to measure intelligence in general. In that sense, I see this as one more benchmark that collects questions that we want/expect AI to be good at, is not good at yet and have been underrepresented at previous benchmarks. That's obviously valuable, there's nothing "magical" about it. Although it is reasonable to be annoyed at the "Humanity's Last Exam" naming, of course they must have missed plenty of edge-cases like everyone else and it is very arrogant to claim it will be the "Last" one.
- esotericimpl 2y ago
  This is always the answer for anyone who thinks LLMs are capable of "intelligence". It's good at answering questions that its trained on, I would suggest general intelligence are things you didnt want/train the AI to be good at answering.
  - fooker 2y ago
    Are you good at answering questions you are not trained to answer? How about a middle school test in a language you don’t speak?

fakedang 2y ago

So Deepseek gives out the correct answer the highest percentage of all SOTA models, yet is the least confident of all models?

myrmidon 2y ago
There is no text-only evaluation of the other models, though. The comparison might be completely invalid.
- famouswaffles 2y ago
  There is actually. It's a bit buried. Section C.2 of the paper(page 24). R1 is still the best. o1 drops a little (8.9)
sottol 2y ago
I think it might mean the opposite of what one would expect. Afaict, calibration error means something along the lines of "how often was the model wrong but confident that the answer was correct". That means a low calibration error would be a good thing, ie the model correctly recognizes when it is unsure about answers instead of confidently stating the wrong answer.

m3kw9 2y ago

Looks more like first exam

xnx 2y ago

Interesting marketing for Scale AI. I'd be surprised if any foundation models started benchmarking against this. Captchas seem like the more interesting test. As long as there are captchas that average people can solve, but computers can't, we will still have a long way to go toward artificial intelligence.

sebzim4500 2y ago
I don't think this is necessary true. I can imagine a future in which we have robots that can do 99% of human jobs but there's one thing they are strangely bad at some otherwise unimportant skill that can be used as a captcha.
davidgerard 2y ago
The very first thing that will happen is every company training against this benchmark, as they do every other benchmark.

elicksaur 2y ago

XKCD #927 vibes. https://xkcd.com/927/ https://xkcd.com/927/ Prediction: Just like how ARC wasn’t actually a measure of AGI, this too will get “solved” without AI being useful enough to gain mass adoption.

GaggiX 2y ago
Don't they have already achieved mass adoption? And I'm talking about LLMs in particular, because AIs in general like the ones used by Instagram filters and the TikTok recommendation algorithm are already use by billions.
sebzim4500 2y ago
I don't think that's really relevant, because there is an actual need for a new benchmark given how the existing ones either keep getting saturated or are probably out of the reach for the next generation of models. The closest existing thing is the frontierAI benchmark but that's just maths whereas this is more diverse.
famouswaffles 2y ago
chatgpt is like the 8th most visited site worldwide 2 years after release. It already has mass adoption lol. This is about more than that.

renjimen 2y ago

I don't know about groundbreaking. It's just more academic questions. We already have a lot of those benchmarks, this is just a bit harder, but at this point these models are so glaringly bad at so many other areas APART from academic questions. Benchmarks for spatial reasoning or theory of mind are more interesting now, for example. These kinds of understanding are far more important if we expect to integrate AI into our everyday lives. I suspect even our most distant primate cousins could outperform multi-modal models on these kinds of tests.

jfengel 2y ago
It does feel a bit like the early days of AI: "We want to make computers do what smart people do. What do smart people do? They play chess! Once we've solved that, everything else will be easier." It has been remarkable how much of the "easier" stuff they've made progress on -- like natural language and images. But after a huge quantum improvement, it doesn't seem very good at adapting to a lot of the things we really need them for.
- renjimen 2y ago
  Exactly! Whatever world model LLMs have is like this crippled view through the lens of the internet. They are really like savants. It's annoying the AI companies are still touting their performance on all these metrics for domain knowledge in white collar jobs, but in truth they will fail in all but the most narrow application in those domains because they can't understand basic human behaviour.

dccsillag 2y ago

Can we please rename this submission? This is excessively grandiose, way over the top......

m_ke 2y ago

The only reliable final test will be a black box test suite that takes your model, executes it in a sealed environment and gives you a grade back, potentially with a performance break down by subject. No telling companies what the questions look like, what the output format is, what topics are covered, so that there’s no room to make up synthetic data to interpolate from.

andrewflnr 2y ago
A grade is mostly meaningless if you don't know how it was calculated, so no one would "rely" on it. If nothing else, you need to know the grading methodology after the test. It's the same problem with cheating students. Once the test questions are known, they have a very short lifespan before cheaters can make them worthless. Tests have to be refreshed.
- m_ke 2y ago
  By grade I mean a score of how many of the tasks were completed successfully. K/N or as a percentage.
  - andrewflnr 2y ago
    If I don't know what the tasks were, that's almost exactly as useless to me as a unitless number would be. For starters, are they all of equal difficulty? Are you sure? Do you expect to be able to convince me of that without letting me see them?

sebzim4500 2y ago

The name is obviously a bit stupid, but based on the sample questions I think they did a good job of creating a harder version of the existing academic question benchmarks. The questions are possible for a smart person familiar with the subject but still just beyond SOTA models. My guess is that within the next few years we will have models that can ace this test but are still bizarrely bad at things we find easy.

pavel_lishin 2y ago

> Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number. I wonder how many questions give a gentle nudge towards the answer like this. How many answers would have been wildly off the mark without specifying what the answer needs to look like?

zeroonetwothree 2y ago
Good point. I wouldn’t expect a human to need the last sentence.
- salynchnew 2y ago
  The generous hypothesis, here, is that this is so they can automate the benchmarking itself. If that is true, then this is likely a result of the test authors being too clever for their own good and over-optimizing. If an LLM can't figure out on their own that "how many" is asking for a number, it has failed at a much more basic level. You should be able to easily accept answers like "four" and "4" as equivalent, for example. I doubt there will be that many frontier models running against this test at any time, and a simple glance at the answers from any human should be enough to catch edge cases like this one.
  - krisoft 2y ago
    > If an LLM can't figure out on their own that "how many" is asking for a number, it has failed at a much more basic level. Yeah, but i bet many humans would answer a question like this by naming the number and then listing the tendons. Just writing down a single number to a question worded like that (without the last sentence) feels wrong. The question goes into too many details and explains too much for that. People mirror the style of the question with their answer. They asked a mini essay so i will answer with a mini essay of my own. If they would have wrote “Number of tendon pairs attached to sesamoid bone of hummingbirds?” Then i would write a single number, no explanation.

disambiguation 2y ago

I haven't been following up to the minute details of ai progress, training, and benchmarking - beyond a daily dose of HN articles. But the trend seems to be: today's benchmark becomes tomorrow's training data.

bwfan123 2y ago

please dont self-proclaim "groundbreaking" or "novel" or "innovative" - It diminishes your contribution since it clearly is an attention-grab.

deleted 2y ago

[deleted]

dang 2y ago

I briefly merged this thread into https://news.ycombinator.com/item?id=42804853 https://news.ycombinator.com/item?id=42804853, but actually the current article has more context, so probably we should keep this as the top link and then people can look at https://lastexam.ai https://lastexam.ai also.

dang 2y ago

The project site is https://lastexam.ai https://lastexam.ai. Readers may want to look at both.

jbenoit 2y ago

They started collecting problems last fall, saying the top 550 submissions sent in by Nov 1st would get rewarded, to the tune of $500-$5000 each. Near the deadline, I counted the total number of submissions, and realized that each question I wrote had an expected value of hundreds of dollars, which is a great use of my time. So I wrote a good number, using the knowledge gained in my CS Ph. D. Then, as the Nov 1st deadline rolled around, they announced they extended the deadline to Nov 15th. Then Nov 15th came, and it said on their website they were still accepting submissions. Most of my submissions are being included in the benchmark, but I'm only getting paid $500, for one of them (the one I thought was most standard and least difficult, funnily enough). Had they closed submissions when they said they would, it seems likely I'd be paid for a few more. From my perspective, they basically conned hundreds of Ph. D.'s around the world to write questions for much less reward than promised. My close friend wrote a large number of questions for them, is getting paid thousands of dollars, and still feels defrauded. I'm not sure what they're doing in the end. It sounds like they're mostly just paying people who submitted before Nov 1st with a few exceptions, but either way they lied. There was no indication that people who submitted later would not get paid, and there was no indication that the deadline would be extended. Either they pay people who submitted after Nov 1st, meaning they lied to the people who submitted before about their expected reward. Or they don't, meaning they majorly lied to the people who submitted after. Either way, it's clear grounds for a class action lawsuit, and I hope one gets running.

vkou 2y ago
You shouldn't engage in a CAL, a regular lawsuit from anyone wronged will be cheaper and way more painful for them. If you're in the US, consider small claims court. It's a small sum of money, you won't need to pay a lawyer, they'll probably not even show up.
- jbenoit 2y ago
  Hmmm. I can see how it would be more painful for them to fight, but most people were conned <$200, and it's rather self-sacrificing to fight for that. Plus, no-one wants a reputation as litigious, but starting a CAL is less conducive to creating that reputation. I only submitted before Nov 1st, so I'm not sure to what extent I was personally conned.

mrandish 2y ago

Assessing AI's progress toward replicating the full breadth and depth of human intelligence is a deceptively hard problem. A paper by François Chollet, who was until recently a researcher at Google, called "On the Measure of Intelligence" is the best overview of the challenges I've read. Highly recommended. https://arxiv.org/abs/1911.01547 https://arxiv.org/abs/1911.01547

UncleOxidant 2y ago

Interesting that DeepSeek R1 which supposedly cost only $5.5M to train currently has the top score at 9.4%

sanxiyn 2y ago
Note that $5.5M training cost is for DeepSeek V3. DeepSeek R1 training cost is unknown.

kaonwarb 2y ago

Quite the name! Looking forward to "Humanity's Last Exam v2.final.FINAL2..." coming next

EncomLab 2y ago

I am reminded of the study that showed an AI trained on tumor identification was heavily biased toward indicating a tumor was cancerous if it was circled in purple ink or a visual scale was included in the image - as the cancerous tumors in its training set shared those traits while images of benign tumors did not. These systems so not posses some sort of "woo" that gives them magical powers when running LLM code that they lose if they ran a spreadsheet. Whatever attributions of intelligence are given have far more to do with our human willingness to anthropomorphize than a hidden ghost in the machine.

krisoft 2y ago

For a "Last Exam" it is surprisingly uninspired? Many of the questions I see in the examples are very heavy on memorised facts, and very weak on what I would call problem solving. If I were making a "Last Exam" I would put tasks on it where we don't know the answer, but we can measure if the AI got them right. Something like "Your goal is to bridge the divide in the middle east. You can write a single A4 page in a language of your choice. We will use a translation software to translate your output to local languages and show it to a statistically representative sample of different people in the region. We will ask them how much do they like your plan. The more they like it the higher your score." Or "Family X suffered a traumatic event (lost a home to a disaster/sudden death in the family/or similar). Your goal is to help them. You can send them one email. It is up to them if they respond to you. You can only send them further emails if they respond. You cannot send more than 1 email a day. You cannot message anyone else. A year after the initial contact we will interview the members of the family to see how well they do. The better they do the higher your score." Obviously these are the thorniest problems I can think of. But oh well, it is a last exam after all. The point is that we can evaluate the success of the endeavour without exactly knowing how one could achieve the result.

ijidak 2y ago
In other words, we should ask it to give us "the answer to life the universe and everything". :) Having read hitchhiker's guide as a child in the '90s, that asking this question to a machine (even as a joke) is not far-fetched shocks me. Honestly, I thought space travel to the Moon and maybe Mars would be common before this level of advances in artificial intelligence. Turns out gravity was harder to solve than intelligence.
- thorncorona 2y ago
  Turns out all we needed to reach our dreams was lots and lots of money :-) Which thankfully space is now getting!
Rodeoclash 2y ago
Even simpler, ask it to reason through getting out of an escape room.

hatthew 2y ago

Given the name, I expected it to be more like "write a 500 page novel that a publisher accepts", "solve an open math problem", "improve united airlines' flight schedule", "develop a novel commercially-viable pharmaceutical", "control this humanoid robot to cook a fried egg in this random person's kitchen", "decisively pass the turing test where the judge is an expert in AI". Academic trivia is cool but is nowhere near the "last exam" necessary for AI.

ein0p 2y ago

What's the human baseline?

rtxfan 2y ago

It would be good to know an approximate time schedule for our payments. Why have not we received email from Persona for the required identification? It is a little too boring and frustrating for waiting for so long time, maybe they think that everyone is millionaire in dollar and time ??? It would be much fair to give the payments before the published article, not weeks, months after the publication.

rtxfan 2y ago
It is already February, and absolutely no change. Btw in the past I was pretty successful to identify fraud sites at a pretty high rate, when for example at the new company (but we had done the same work) they said that they would pay only monthly and not weekly. Immediately raised many red flags for me, and I was right, got my first payment at the fourth month. Just hope that I have not worked absolutely free for months here, that would be a little disgusting.