4 ms·

A bad article, it test the visual acuity of ChatGPT, correctly estimate that it may just have a "bad eyesight", then ask it to sum different numbers, a difficul

by Kuinox 3y ago

A bad article, it test the visual acuity of ChatGPT, correctly estimate that it may just have a "bad eyesight", then ask it to sum different numbers, a difficult problem for ChatGPT since it see everything in tokens.

dkjaudyeqooe 3y ago
So it's not intelligent, just a token processing automation.
- huytersd 3y ago
  That depends. Eyesight doesn’t mean intelligence. It would be like evaluating the intelligence of a mostly blind individual on a standard IQ test. How does it perform if we can provide the test in a form it can process. Vector images or something similar maybe.
- Kuinox 3y ago
  I'm saying the author just showed it have a bad eyesight, then said it's thinking ability is low because it can't do sums, if you reformulate the exercise in a way it can ingest, and something that make sense for how it understand data, it can complete the exercises: Here with my prompt: xoo ooo ooo oxo ooo ooo oox ooo ooo ooo xoo ooo ooo oxo ooo ooo oox ooo ooo ooo xoo ooo ooo oxo ??? ??? ??? What the "???" should be ? Answer A: ooo ooo oox Answer B: ooo ooo xoo Answer C: ooo ooo oxo Answer D: ooo oox ooo Answer E: oxo ooo ooo Answer F: ooo oxo ooo Mistral Large: > Based on the pattern, it appears that the "x" is moving one step to the right in each row, then moving down to the next row and starting again from the leftmost position. Given this, the "???" should be: ooo ooo oox So, the correct answer is A. Edit: note it doesn't always respond correctly. I ran it 10 times on mistral large, and it responded correctly 8 times. In the two time it failed, it correctly describe what the final sequence is, but it selected the incorrect response.
  - FergusArgyll 3y ago
    fwiw: Gemini gets this wrong! The answer is most likely Answer B: ooo ooo xoo. Here's why: The pattern repeats every three rows. The first three rows establish a pattern of "x" alternating between the first and third columns. "o" fills the remaining spaces. Looking at the last two sets of three rows, we see ooo xoo ooo followed by ooo ooo oxo. Following the established pattern, the next set of three rows should be ooo ooo xoo, completing the cycle. While other answers might fit in a single instance, they wouldn't maintain the consistent pattern throughout the entire sequence.
    - Kuinox 3y ago
      I canceled my ChatGPT subscription recently and I can't test it on on GPT4 either. Mistral Large should be inferior to GPT4 according to mistral own benchmark. Also, look at my edit, don't run it only once, LLMs arent infered deterministically and it can give you a wrong response even if it respond correctly 80% of the times.
      - fauigerzigerk 3y ago
        I tried GPT-4 with my modified prompt (see https://news.ycombinator.com/item?id=39583414 https://news.ycombinator.com/item?id=39583414). Here's the response: The patterns provided show a progressive movement of "x" in a clockwise direction starting from the top-left corner, moving to the middle of the top row, then to the top-right corner, descending down to the middle of the right column, moving further down to the bottom-right corner, then to the middle of the bottom row, and finally to the bottom-left corner. Following this sequence, the "x" would move to the middle of the left column after the bottom-left corner, which is represented by: o x o However, none of the provided answer choices directly match this progression, as they all consist of 3x3 grids with "x" in various positions. Given the pattern's progression, after reaching the bottom-left position, the next logical step would be to move back to the initial position or follow the pattern as closely as possible within the provided options. Since the last illustrated pattern before the question marks is: ooo ooo oxo The correct substitution that follows this clockwise movement would be to place "x" back at the top-left corner, as the cycle restarts. However, this direct option isn't provided. The closest next step according to the given pattern progression would actually be placing "x" in the bottom-right corner to continue the clockwise motion: ooo ooo oox Thus, the correct substitution for the last pattern with the given options would be: Answer A: ooo ooo oox This choice follows the logical progression of the pattern as "x" moves clockwise around the perimeter of the 3x3 grid.
jquery 3y ago
It would be like giving a human an iq test based on smells.
- Kuinox 3y ago
  Smells are very culture dependents, some culture have vocabulary to describe parts of the smell that allow the brains to remember more accuratly a smell.
NovemberWhiskey 3y ago
If you're not intelligent, then indeed tests designed to test intelligence may seem to represent difficult problems.
- Kuinox 3y ago
  When you do an IQ test, the person testing take into account the handicap of the person. I redid a prompt and ran it on Mistral Large and it answered without any problem, see my comment below.

by Kuinox 3y ago

dkjaudyeqooe 3y ago

So it's not intelligent, just a token processing automation.

huytersd 3y ago
That depends. Eyesight doesn’t mean intelligence. It would be like evaluating the intelligence of a mostly blind individual on a standard IQ test. How does it perform if we can provide the test in a form it can process. Vector images or something similar maybe.
Kuinox 3y ago
I'm saying the author just showed it have a bad eyesight, then said it's thinking ability is low because it can't do sums, if you reformulate the exercise in a way it can ingest, and something that make sense for how it understand data, it can complete the exercises: Here with my prompt: xoo ooo ooo oxo ooo ooo oox ooo ooo ooo xoo ooo ooo oxo ooo ooo oox ooo ooo ooo xoo ooo ooo oxo ??? ??? ??? What the "???" should be ? Answer A: ooo ooo oox Answer B: ooo ooo xoo Answer C: ooo ooo oxo Answer D: ooo oox ooo Answer E: oxo ooo ooo Answer F: ooo oxo ooo Mistral Large: > Based on the pattern, it appears that the "x" is moving one step to the right in each row, then moving down to the next row and starting again from the leftmost position. Given this, the "???" should be: ooo ooo oox So, the correct answer is A. Edit: note it doesn't always respond correctly. I ran it 10 times on mistral large, and it responded correctly 8 times. In the two time it failed, it correctly describe what the final sequence is, but it selected the incorrect response.
- FergusArgyll 3y ago
  fwiw: Gemini gets this wrong! The answer is most likely Answer B: ooo ooo xoo. Here's why: The pattern repeats every three rows. The first three rows establish a pattern of "x" alternating between the first and third columns. "o" fills the remaining spaces. Looking at the last two sets of three rows, we see ooo xoo ooo followed by ooo ooo oxo. Following the established pattern, the next set of three rows should be ooo ooo xoo, completing the cycle. While other answers might fit in a single instance, they wouldn't maintain the consistent pattern throughout the entire sequence.
  - Kuinox 3y ago
    I canceled my ChatGPT subscription recently and I can't test it on on GPT4 either. Mistral Large should be inferior to GPT4 according to mistral own benchmark. Also, look at my edit, don't run it only once, LLMs arent infered deterministically and it can give you a wrong response even if it respond correctly 80% of the times.
    - fauigerzigerk 3y ago
      I tried GPT-4 with my modified prompt (see https://news.ycombinator.com/item?id=39583414 https://news.ycombinator.com/item?id=39583414). Here's the response: The patterns provided show a progressive movement of "x" in a clockwise direction starting from the top-left corner, moving to the middle of the top row, then to the top-right corner, descending down to the middle of the right column, moving further down to the bottom-right corner, then to the middle of the bottom row, and finally to the bottom-left corner. Following this sequence, the "x" would move to the middle of the left column after the bottom-left corner, which is represented by: o x o However, none of the provided answer choices directly match this progression, as they all consist of 3x3 grids with "x" in various positions. Given the pattern's progression, after reaching the bottom-left position, the next logical step would be to move back to the initial position or follow the pattern as closely as possible within the provided options. Since the last illustrated pattern before the question marks is: ooo ooo oxo The correct substitution that follows this clockwise movement would be to place "x" back at the top-left corner, as the cycle restarts. However, this direct option isn't provided. The closest next step according to the given pattern progression would actually be placing "x" in the bottom-right corner to continue the clockwise motion: ooo ooo oox Thus, the correct substitution for the last pattern with the given options would be: Answer A: ooo ooo oox This choice follows the logical progression of the pattern as "x" moves clockwise around the perimeter of the 3x3 grid.

jquery 3y ago

It would be like giving a human an iq test based on smells.

Kuinox 3y ago
Smells are very culture dependents, some culture have vocabulary to describe parts of the smell that allow the brains to remember more accuratly a smell.

NovemberWhiskey 3y ago

If you're not intelligent, then indeed tests designed to test intelligence may seem to represent difficult problems.

Kuinox 3y ago
When you do an IQ test, the person testing take into account the handicap of the person. I redid a prompt and ran it on Mistral Large and it answered without any problem, see my comment below.