8 ms·

Ok maybe I should rephrase but think of it like this: Would your IQ test score be accurate if you had a week beforehand to train solving IQ tests? You don't g

Ok maybe I should rephrase but think of it like this: Would your IQ test score be accurate if you had a week beforehand to train solving IQ tests? You don't get exactly the same test in the end, similar with SAT, but the constraints we put on these tests (they have to be comparable) produce patterns in the questions you can train for. This is the same logic why people can train to improve their SAT scores, if they were a measure of true innate intelligence their training would have no impact on their score.

siva7 4y ago
I mean, let's be fair. That's also how many humans learn and do IQ tests.
- prng2021 4y ago
  Yes but the critical difference is that AI couldn’t pass these tests if they weren’t trained on a very similar set of questions and answers. Not every person takes a SAT prep class to improve their test score. There are lots of people who are truly above average in terms of intelligence and can score very high on the first try.
  - naasking 4y ago
    > Yes but the critical difference is that AI couldn’t pass these tests if they weren’t trained on a very similar set of questions and answers. First, I don't think that's strictly true. Obviously they wouldn't do as well but they still do better than chance. Second, there's evidence that this is a big part of the Flynn effect, which means humans are susceptible to a similar phenomenon.
    - prng2021 4y ago
      “ Obviously they wouldn't do as well but they still do better than chance” is a huge understatement. If a model wasn’t trained with any SAT questions and answers but was trained with the same verbal and mathematical knowledge a high school student would have, the AI would do extremely poorly in an actual test. In contrast, the vast majority of human test takers would score leagues above picking answers by chance. Your original reply insinuated that the AI is learning very similar to how humans do and that’s just not true. Yes, humans do pattern matching based on prior experiences/knowledge like AI does when you train a model, but human intelligence goes way beyond that.
      - naasking 4y ago
        > Yes, humans do pattern matching based on prior experiences/knowledge like AI does when you train a model, but human intelligence goes way beyond that. Humans are trained on orders of magnitude more multimodal data over their lifetimes. Also, humans are not borne as an unbiased model, billions of years of evolution have crafted many implicit biases into our cognition (like a propensity to language, facial recognition, etc.). All machine learning models are true blank slates, so it takes a lot more data just to build up to the same starting point as a newborn human. All that's to say that you have no basis upon which to claim that AI learning is NOT similar to how humans do it, or that human intelligence "goes way beyond hat", it's just that humans have a head start and a lot more data to work with.
        prng2021 4y ago
        “Humans are trained on orders of magnitude more multimodal data over their lifetimes. Also, humans are not borne as an unbiased model, billions of years of evolution have crafted many implicit biases into our cognition (like a propensity to language, facial recognition, etc.).” What in the world are you talking about? I must be talking to chatgpt and am done with this thread. We were originally discussing the differences in methodology between AI and humans for passing standardized exams. Those involve tasks like applying well-defined mathematical concepts to a brand new problem, not “multimodal” data or facial recognition.
        naasking 4y ago
        If you don't understand what I'm talking about then you don't understand how how these transformer AIs learn and solve problems, so maybe you shouldn't opine about how AI couldn't pass these tests. Your claimed differences between how humans and AIs work is conjecture that can be explained by what I described rather than fundamental differences in how these systems work.
        d0mine 4y ago
        Here's simple example: (x-x+c)=c gpt struggles with such examples if `x` is a large number e.g., x=123_000_000_456 and `c` some specific number, but it is easy for humans.
codethief 4y ago
But hasn't that always been a problem inherent to IQ tests (and SAT tests, of course)?
- beepbooptheory 4y ago
  It is definitely one of the problems you add to a pile marked "IQ tests are meaningless and cater to the worst conceptions of human intelligence."
amoss 4y ago
If we flip the question around it seems more reasonable: would your IQ test score be accurate if you did not train beforehand on practice IQ tests? I would presume not - most tests are timed, and if you are spending time on first-time only tasks in understanding the problem then the result is inaccurate. If you train out those first-time tasks so that you are repeatably using the time budget in the test to solve problems then you should reach some kind of steady state and produce repeatable and more accurate test scores. My take is that the repeatable scores measuring your steady state in the task would be more accurate than the untrained scores with an unknown amount of initialization time within each problem. I would make a similar claim to naasking below that this could account for some of the Flynn effect.
adamsmith143 4y ago
> This is the same logic why people can train to improve their SAT scores So isn't this literally moving the goalpost? "So what an AI can beat the SAT, so can humans"

siva7 4y ago

I mean, let's be fair. That's also how many humans learn and do IQ tests.

prng2021 4y ago
Yes but the critical difference is that AI couldn’t pass these tests if they weren’t trained on a very similar set of questions and answers. Not every person takes a SAT prep class to improve their test score. There are lots of people who are truly above average in terms of intelligence and can score very high on the first try.
- naasking 4y ago
  > Yes but the critical difference is that AI couldn’t pass these tests if they weren’t trained on a very similar set of questions and answers. First, I don't think that's strictly true. Obviously they wouldn't do as well but they still do better than chance. Second, there's evidence that this is a big part of the Flynn effect, which means humans are susceptible to a similar phenomenon.
  - prng2021 4y ago
    “ Obviously they wouldn't do as well but they still do better than chance” is a huge understatement. If a model wasn’t trained with any SAT questions and answers but was trained with the same verbal and mathematical knowledge a high school student would have, the AI would do extremely poorly in an actual test. In contrast, the vast majority of human test takers would score leagues above picking answers by chance. Your original reply insinuated that the AI is learning very similar to how humans do and that’s just not true. Yes, humans do pattern matching based on prior experiences/knowledge like AI does when you train a model, but human intelligence goes way beyond that.
    - naasking 4y ago
      > Yes, humans do pattern matching based on prior experiences/knowledge like AI does when you train a model, but human intelligence goes way beyond that. Humans are trained on orders of magnitude more multimodal data over their lifetimes. Also, humans are not borne as an unbiased model, billions of years of evolution have crafted many implicit biases into our cognition (like a propensity to language, facial recognition, etc.). All machine learning models are true blank slates, so it takes a lot more data just to build up to the same starting point as a newborn human. All that's to say that you have no basis upon which to claim that AI learning is NOT similar to how humans do it, or that human intelligence "goes way beyond hat", it's just that humans have a head start and a lot more data to work with.
      - prng2021 4y ago
        “Humans are trained on orders of magnitude more multimodal data over their lifetimes. Also, humans are not borne as an unbiased model, billions of years of evolution have crafted many implicit biases into our cognition (like a propensity to language, facial recognition, etc.).” What in the world are you talking about? I must be talking to chatgpt and am done with this thread. We were originally discussing the differences in methodology between AI and humans for passing standardized exams. Those involve tasks like applying well-defined mathematical concepts to a brand new problem, not “multimodal” data or facial recognition.
        naasking 4y ago
        If you don't understand what I'm talking about then you don't understand how how these transformer AIs learn and solve problems, so maybe you shouldn't opine about how AI couldn't pass these tests. Your claimed differences between how humans and AIs work is conjecture that can be explained by what I described rather than fundamental differences in how these systems work.
        d0mine 4y ago
        Here's simple example: (x-x+c)=c gpt struggles with such examples if `x` is a large number e.g., x=123_000_000_456 and `c` some specific number, but it is easy for humans.

codethief 4y ago

But hasn't that always been a problem inherent to IQ tests (and SAT tests, of course)?

beepbooptheory 4y ago
It is definitely one of the problems you add to a pile marked "IQ tests are meaningless and cater to the worst conceptions of human intelligence."

amoss 4y ago

If we flip the question around it seems more reasonable: would your IQ test score be accurate if you did not train beforehand on practice IQ tests? I would presume not - most tests are timed, and if you are spending time on first-time only tasks in understanding the problem then the result is inaccurate. If you train out those first-time tasks so that you are repeatably using the time budget in the test to solve problems then you should reach some kind of steady state and produce repeatable and more accurate test scores. My take is that the repeatable scores measuring your steady state in the task would be more accurate than the untrained scores with an unknown amount of initialization time within each problem. I would make a similar claim to naasking below that this could account for some of the Flynn effect.

adamsmith143 4y ago

> This is the same logic why people can train to improve their SAT scores So isn't this literally moving the goalpost? "So what an AI can beat the SAT, so can humans"