3 ms·
> improved factuality I assume those release notes apply to when I tried ChatGPT for the first time yesterday. One of my prompts was asking, both in English an
by glandium 4y ago
> improved factuality
I assume those release notes apply to when I tried ChatGPT for the first time yesterday. One of my prompts was asking, both in English and in Japanese, who the Japanese Prime Minister was. I made it regenerate an answer 3 times for each.
In English, it consistently gave acurate information for the time it said it had data for (2021), albeit formulated slightly differently each time, but every time including the name of the Prime Minister (Suga), the date he started (Sep 16, 2020), his political affiliation (LDP), and that he was Chief Cabinet Secretary under Abe before that.
In Japanese, however, it went all over the place. First time, it gave the right name, saying it was current as of August 2021, and that he started on Dec 28, 2021 (not only is the date wrong, it's past the "as of", and funnily enough, Suga was out of office already by then (I had picked that specifically because Japanese Prime Ministers are volatile)). It then went on with a mix of good and wrong information. The second time, it was saying the PM was Mori Yoshiro (who was PM... 22 years ago, but it was still saying "as of 2021"). Third time, it switched to Abe Shinzo, and the fourth time, to Hatoyama Yukio. All former PMs, but it was clearly saying they were current as of 2021. In the last case it was also saying that he had been PM already in 2009-2010 (which is true), implying it was his second time as PM (which is false).
What this suggests to me is that it doesn't have any global knowledge, that it can regurgitate in different languages, but that it has some contextualized knowledge where the language is part of the context, but the amount of data it was fed in various languages differ, and the amount of e.g. Japanese context it has might not be enough.
Interestingly, though, the way it writes Japanese, where e.g. it adds a subject to sentences where natives wouldn't, does feel like it could be translating from another language.
At some point, I informed it that Abe was killed this year, to which it replied that it's not true but that it could be missing information, inviting me to provide more, which I did, after which it assertively told me Abe was very well alive, while still adding that it might have some information missing...
- londons_explore 4y agoChatGPT's training process of ingesting many articles with a context window of a few thousand words means it often can't see the date that an article or bit of text was written. So as far as it's concerned, when learning from an article written in 2006, the prime minister of Japan really is Abe right now. I suspect this could be fixed by having an extra training input of 'date this text was written', and then when generating an output, specify that you want the output to be written as-if the writer is writing it in a specific year.
- londons_explore 4y agoDespite this being called the Jan 9th update, I don't think it was actually live yesterday.