3 ms·

He said nothing controversial and yes the model has digressed. This "flagship model" is fast and cheaper, but it is worse (regardless of benchmarks & charts, su

by ramoz 2y ago

He said nothing controversial and yes the model has digressed. This "flagship model" is fast and cheaper, but it is worse (regardless of benchmarks & charts, subjectivity is failing for me and many others). The model seems better at logic, but something is off. It is repetitive of the user prompt, gets hooked on logical meaning vs contextual (per provided conversation), and is too concise as if it needs to provide a direct answer. Perhaps all good for a benchmark, code, or simple logic flows/chains (e.g. ai validation - which gpt-3.5 is good at as well). But it's not desirable for advanced/creative content generation. My anecdotal receipt: None of the 15 functions we run in prod are switching to the 50% off model as product quality is degraded with gpt-4o.

lioeters 2y ago
> it is worse..regardless of benchmarks & charts It's interesting how this is a fairly common sentiment expressed, at least I heard it often for previous models. Do you think such weaknesses can be quantified by adding more test cases, and improving the benchmarks? Maybe it would help if they opened up to community contributions, so people could submit test cases that demonstrate issues that they (OpenAI, et al) are currently not seeing.
pieix 2y ago
> My anecdotal receipt: None of the 15 functions we run in prod are switching to the 50% off model as product quality is degraded with gpt-4o. Interesting anecdata — how did you validate this?
- ramoz 2y ago
  My personal case is a flow/chain of many function calls for creative output. Flipping the switch resulted in noticeably different and undesired results. We tested a ton. This tweeter normally has a narrative I turn away from, but the thread was useful as it at least made me feel like I wasn’t losing my mind by finding similar accounts. https://x.com/bindureddy/status/1790127425705120149?s=46&t=yz7rONTEj7dbvr476o5CCw https://x.com/bindureddy/status/1790127425705120149?s=46&t=y...

by ramoz 2y ago

lioeters 2y ago

> it is worse..regardless of benchmarks & charts It's interesting how this is a fairly common sentiment expressed, at least I heard it often for previous models. Do you think such weaknesses can be quantified by adding more test cases, and improving the benchmarks? Maybe it would help if they opened up to community contributions, so people could submit test cases that demonstrate issues that they (OpenAI, et al) are currently not seeing.

pieix 2y ago

> My anecdotal receipt: None of the 15 functions we run in prod are switching to the 50% off model as product quality is degraded with gpt-4o. Interesting anecdata — how did you validate this?

ramoz 2y ago
My personal case is a flow/chain of many function calls for creative output. Flipping the switch resulted in noticeably different and undesired results. We tested a ton. This tweeter normally has a narrative I turn away from, but the thread was useful as it at least made me feel like I wasn’t losing my mind by finding similar accounts. https://x.com/bindureddy/status/1790127425705120149?s=46&t=yz7rONTEj7dbvr476o5CCw https://x.com/bindureddy/status/1790127425705120149?s=46&t=y...