5 ms·

I played along the example of the chancellor question. If you deviate a little from the examples in the article GPT-4 gets it all wrong: Who is the eighth Fed

I played along the example of the chancellor question. If you deviate a little from the examples in the article GPT-4 gets it all wrong: Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel) https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f9de3f2 https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f... Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Helmut Kohl (wrong, Angela Merkel) https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f9de3f2 https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f... Interestingly whether you put is or was into the question does make a difference. Even when allowed to surf the web, ChatGPT gets it wrong: Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder (wrong, Angela Merkel) https://chat.openai.com/share/867cb3bc-f642-4d2a-b80d-fe8dc00eef3c https://chat.openai.com/share/867cb3bc-f642-4d2a-b80d-fe8dc0... Though it's referring to the right Wikipedia page: https://en.m.wikipedia.org/wiki/List_of_chancellors_of_Germany https://en.m.wikipedia.org/wiki/List_of_chancellors_of_Germa...

dragonwriter 3y ago
> Even when allowed to surf the web, ChatGPT gets it wrong: > Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder Its probably counting Walter Scheel, who was Acting Chancellor in 1974, and you are probably not. When I asked ChatGPT to list the chancellors in order and identify the eighth, it listed eight ending in Schröder, with Scheel and his ten day Acting Chancellor tenure in May 74 as number 5. (Scheel’s tenure is in the timeline on the Wikipedia page you cite, though not in the numbered list in that page, which is why there is a dicontinuity in thr dates on the numbered list.)
- littlestymaar 3y ago
  It's funny because you ascribe some reasoning in ChatGPT's answer “Gerhard Schröder”, but somehow missed that it's also able to give you two other answers that are unambiguously wrong…
  - dragonwriter 3y ago
    > It's funny because you ascribe some reasoning in ChatGPT's answer “Gerhard Schröder”, but somehow missed that it's also able to give you two other answers that are unambiguously wrong Schröder is the only answer it gave when using functionality that would bring some representation if a list into its context first, and 10it did it with different prompts and mechanisms for bringing a list into its context. That LLMs are bad at counting-related tasks without doing that is well-known, and not a point I felt needed belaboring.
- postexitus 3y ago
  If you think ChatGPT is "counting" anything, you have massively missed the point.
- RandomLensman 3y ago
  Walter Scheel was never Chancellor, he was fulfilling the duties of the Chancellor while being Foreign Minister, but he was not a Chancellor. So counting him is wrong.
  - Kiro 3y ago
    Imagine reading this comment thread a few years back. Just getting an answer to this question at all from a generic bot would be considered science-fiction and now we're dismissing it based on technicalities.
    - RandomLensman 3y ago
      Not a technicality. It's like inventing a proper head of government that wasn't there. Also, knowing German chancellors was always easy, so that fact that a machine falls behind any normal dictionary or the German Chancellor's website is poor form. It's like a calculator getting basic addition wrong. I don't want a future full of poorly performing machines.
      - Kiro 3y ago
        He was acting Chancellor. It's like saying a red apple is not an apple. It's a technicality. The fact that we have a completely reasonable explanation to why GPT-4 includes him in the list and you guys still make it sound like the AI is a complete moron is just hilarious to me.
        RandomLensman 3y ago
        There is no "Acting Chancellor", that is a poor translation of what the job is. There is only a Chancellor, not an "Acting Chancellor" - you have to trust that the Germans and their institutions kind of know who was Chancellor and who was not. The machine is a complete moron for not being able to get basic data from basic sources right.
        3y ago
carlmr 3y ago
>Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel) Technically there's no right answer to this question. "Is" implies present. But Angela Merkel isn't the present chancellor. Olaf Scholz is. But he's not the eigth.
- pwdisswordfishc 3y ago
  Technically, Angela Merkel never stopped being the person who served as the eighth chancellor.
- amelius 3y ago
  Language is imprecise. LLMs should be able to deal with that.
zurfer 3y ago
as others pointed out, it's important how you ask and what you think the truth is. https://chat.openai.com/share/cc688592-236a-499b-82e1-81bdaaa88afe https://chat.openai.com/share/cc688592-236a-499b-82e1-81bdaa...
moelf 3y ago
your first two links are duplicates
getpost 3y ago
I asked for a list of Federal Chancellors of the Federal Republic of Germany and GPT4 included Scheel ("acting"), which makes Schröder the 8th. I often troubleshoot GPT4 replies by asking it for more context. "Is" vs "was" is interesting an find! https://chat.openai.com/share/ddd2800e-630c-4ff7-9992-14b265f41eb3 https://chat.openai.com/share/ddd2800e-630c-4ff7-9992-14b265...

dragonwriter 3y ago

> Even when allowed to surf the web, ChatGPT gets it wrong: > Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder Its probably counting Walter Scheel, who was Acting Chancellor in 1974, and you are probably not. When I asked ChatGPT to list the chancellors in order and identify the eighth, it listed eight ending in Schröder, with Scheel and his ten day Acting Chancellor tenure in May 74 as number 5. (Scheel’s tenure is in the timeline on the Wikipedia page you cite, though not in the numbered list in that page, which is why there is a dicontinuity in thr dates on the numbered list.)

littlestymaar 3y ago
It's funny because you ascribe some reasoning in ChatGPT's answer “Gerhard Schröder”, but somehow missed that it's also able to give you two other answers that are unambiguously wrong…
- dragonwriter 3y ago
  > It's funny because you ascribe some reasoning in ChatGPT's answer “Gerhard Schröder”, but somehow missed that it's also able to give you two other answers that are unambiguously wrong Schröder is the only answer it gave when using functionality that would bring some representation if a list into its context first, and 10it did it with different prompts and mechanisms for bringing a list into its context. That LLMs are bad at counting-related tasks without doing that is well-known, and not a point I felt needed belaboring.
postexitus 3y ago
If you think ChatGPT is "counting" anything, you have massively missed the point.
RandomLensman 3y ago
Walter Scheel was never Chancellor, he was fulfilling the duties of the Chancellor while being Foreign Minister, but he was not a Chancellor. So counting him is wrong.
- Kiro 3y ago
  Imagine reading this comment thread a few years back. Just getting an answer to this question at all from a generic bot would be considered science-fiction and now we're dismissing it based on technicalities.
  - RandomLensman 3y ago
    Not a technicality. It's like inventing a proper head of government that wasn't there. Also, knowing German chancellors was always easy, so that fact that a machine falls behind any normal dictionary or the German Chancellor's website is poor form. It's like a calculator getting basic addition wrong. I don't want a future full of poorly performing machines.
    - Kiro 3y ago
      He was acting Chancellor. It's like saying a red apple is not an apple. It's a technicality. The fact that we have a completely reasonable explanation to why GPT-4 includes him in the list and you guys still make it sound like the AI is a complete moron is just hilarious to me.
      - RandomLensman 3y ago
        There is no "Acting Chancellor", that is a poor translation of what the job is. There is only a Chancellor, not an "Acting Chancellor" - you have to trust that the Germans and their institutions kind of know who was Chancellor and who was not. The machine is a complete moron for not being able to get basic data from basic sources right.
        3y ago

carlmr 3y ago

>Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel) Technically there's no right answer to this question. "Is" implies present. But Angela Merkel isn't the present chancellor. Olaf Scholz is. But he's not the eigth.

pwdisswordfishc 3y ago
Technically, Angela Merkel never stopped being the person who served as the eighth chancellor.
amelius 3y ago
Language is imprecise. LLMs should be able to deal with that.

zurfer 3y ago

as others pointed out, it's important how you ask and what you think the truth is. https://chat.openai.com/share/cc688592-236a-499b-82e1-81bdaaa88afe https://chat.openai.com/share/cc688592-236a-499b-82e1-81bdaa...

moelf 3y ago

your first two links are duplicates

getpost 3y ago

I asked for a list of Federal Chancellors of the Federal Republic of Germany and GPT4 included Scheel ("acting"), which makes Schröder the 8th. I often troubleshoot GPT4 replies by asking it for more context. "Is" vs "was" is interesting an find! https://chat.openai.com/share/ddd2800e-630c-4ff7-9992-14b265f41eb3 https://chat.openai.com/share/ddd2800e-630c-4ff7-9992-14b265...