3 ms·

I read the paper and while I agree that maybe math is not GPT-4 strongest point, I have noticed the same degradation in quality. And as I know there will be ma

by tagyro 3y ago

I read the paper and while I agree that maybe math is not GPT-4 strongest point, I have noticed the same degradation in quality. And as I know there will be many commenters asking, here's my experience. Four months ago I released a mobile app that wraps the "Open"AI API and allows a more private use. The app also includes 15 domains with over 150+ editable prompts, specifically crafted to help users get the most out of GPT. I initially crafted these prompts in English but given I wanted to allow also non-English speakers to make use of them, I started to translate them in Italian and German. I wrote a small script that took each English prompt and, with some more prompt-fu, translated them to these languages, using GPT-4. As I'm fairly fluent in both Italian and German, I was able to verify the quality of these translations. Well, around end of May, when I first noticed some weird answers from GPT, I ran my script again and (surprise!) the quality of the translations is visible inferior. On a different note, I noticed that from one day to the other, any prompt would get an empty reply in the app, only to discover that "Open"AI has also made a subtle change in the json format of the API response that made broke the parser in the app. Admittedly, 150 phrases is not a huge sample size and I could/should have used the default json parser. I don't trust "Open"AI and will, as soon as it's feasible, change to a different model.

nmfisher 3y ago
This tracks with my experience too. For the past few years, I've been using "give me the pinyin for these Chinese characters" as a rough guide for how well a given model would perform. When GPT4 was released, it absolutely nailed this task; every word was 100% correct, every single time. For reference, even GPT3.5 wasn't great at this. I repeated this last week and it's still getting the phonetic content right, it's now often getting the tones wrong (something which I literally never saw on release). Perhaps changing the prompt would deliver the correct results, but that's not really the point. Something has pretty clearly changed such that old prompts no longer delivering the same results.

by tagyro 3y ago

nmfisher 3y ago

This tracks with my experience too. For the past few years, I've been using "give me the pinyin for these Chinese characters" as a rough guide for how well a given model would perform. When GPT4 was released, it absolutely nailed this task; every word was 100% correct, every single time. For reference, even GPT3.5 wasn't great at this. I repeated this last week and it's still getting the phonetic content right, it's now often getting the tones wrong (something which I literally never saw on release). Perhaps changing the prompt would deliver the correct results, but that's not really the point. Something has pretty clearly changed such that old prompts no longer delivering the same results.