5 ms·

This is incorrect. If you take the most basic interpretation of an LLM at temperature 0 as predicting the most likely token, and you run it on, say, 1,000 runs

This is incorrect. If you take the most basic interpretation of an LLM at temperature 0 as predicting the most likely token, and you run it on, say, 1,000 runs of "complete this Spanish sentence with the word for 'X'", then: - maybe ALL humans would fail the test in some way, eg. let's say everybody gets at least 10 of those wrong, and the average person gets 100 of those wrong. - still, as long as most people correctly get each word right, your LLM would get every single response correct (because for each item in the test, 900+ people out of a thousand gave the same correct answer in the training set). In that sense, it's totally possible for a system trained on a vast vat of average-human input to generate super-human outputs.

sambapa 2y ago
But still, the questions in that test are "solved" in the sense of "I can take a dictionary and answers these questions with full certainty". Beyond established knowledge LLMs are monkeys with typewriters, at best.
- yunwal 2y ago
  I’d like to see you ace even a middle-school level Spanish test with just a dictionary (sub Spanish with some other language if you happen to know Spanish).
  - sambapa 2y ago
    It was a figure of speech. But there is nothing superintelligent about acing Spanish tests. Give me a Riemann hypothesis.
    - sebastiennight 2y ago
      Let's define "zeta" as a mathematical function ζ(s) which takes a statement "s" as input, where s is a statement of a breakthrough in LLM capabilities achieved relative to the current date and time and ζ(s) is the probability that a given AI skeptic will honestly recognize "s" as a breakthrough, then our Riemann-Goalpost hypothesis is that ζ has zeros for every "s" which is a negative integer (every breakthrough that happened in the past is null in value) and only has positive values where s is positive. We can conclude from the above that given a far enough date in the future, any given breakthrough can be spectacular, but once achieved, will be derided as trivial.

sambapa 2y ago

But still, the questions in that test are "solved" in the sense of "I can take a dictionary and answers these questions with full certainty". Beyond established knowledge LLMs are monkeys with typewriters, at best.

yunwal 2y ago
I’d like to see you ace even a middle-school level Spanish test with just a dictionary (sub Spanish with some other language if you happen to know Spanish).
- sambapa 2y ago
  It was a figure of speech. But there is nothing superintelligent about acing Spanish tests. Give me a Riemann hypothesis.
  - sebastiennight 2y ago
    Let's define "zeta" as a mathematical function ζ(s) which takes a statement "s" as input, where s is a statement of a breakthrough in LLM capabilities achieved relative to the current date and time and ζ(s) is the probability that a given AI skeptic will honestly recognize "s" as a breakthrough, then our Riemann-Goalpost hypothesis is that ζ has zeros for every "s" which is a negative integer (every breakthrough that happened in the past is null in value) and only has positive values where s is positive. We can conclude from the above that given a far enough date in the future, any given breakthrough can be spectacular, but once achieved, will be derided as trivial.