3 ms·
I’m still waiting to see even mildly convincing evidence of this supposed change. I’m no shill, and it’s certainly in the realm of things OpenAI might do - but
by ShamelessC 3y ago
I’m still waiting to see even mildly convincing evidence of this supposed change. I’m no shill, and it’s certainly in the realm of things OpenAI might do - but if it’s truly occurring and the degradation is enough to anger people; at least one person should be able to present prompts and responses from both before and after, using a range of different temperatures and performing multiple runs for each prompt.
The idea that not even a single GPT user thought to run this sort of evaluation (and particularly businesses that depend on the quality remaining consistent) says to me that people just didn’t understand the flaws of the model until more prolonged use.
I’m happy to be corrected of course, but I’d rather not hear another one-off anecdote without any sort of rigor applied.
- chaxor 3y agoI don't know what is true either, but you can't run that analysis for the claim people are making. The reason being that people aren't claiming the API model outputs changed, they're claiming the website based outputs changed. Most claim it's probably due to some alteration in the 'pre-prompt', the hyperparameters (temp, top k instead of typical sampling, etc) or something to that effect. But you can't really test that so easily with a non deterministic temp.
- bun_at_work 3y agoYou asking how people know the models being used have changed? If you use the OpenAI APIs to access these models the specific version used is kicked back in the response. Also: https://platform.openai.com/docs/models/continuous-model-upgrades https://platform.openai.com/docs/models/continuous-model-upg... I'm only speaking about OpenAI here, not Bard.
- ShamelessC 3y agoNo, I’m referring to the rumors of OpenAI quietly changing the model and then publicly claiming they remain static per version (as your link indicates).
- ravenstine 3y agoMaybe some enterprising HN'er needs to build a "gpttracker.com" that regularly tests OpenAI's models against a standard set of prompts to reveal whether significant updates are being made and at what rate.
- pclmulqdq 3y agoThree days after that gets released, all the problem prompts will be fixed, and it will have a new set of flaws. OpenAI is known to do that.