4 ms·

> other models are in the ballpark How true is this? From playing around with Bard and Claude, GPT-4 seems to be significantly better, especially around code g

> other models are in the ballpark How true is this? From playing around with Bard and Claude, GPT-4 seems to be significantly better, especially around code generation / understanding.

letitgo12345 4y ago
People really don't seem to understand just how far ahead OAI is in its reasoning abilities (https://crfm.stanford.edu/helm/v0.2.2/?group=reasoning https://crfm.stanford.edu/helm/v0.2.2/?group=reasoning) Maybe PaLM is near there (it's not evaluated on that page) but nothing else even comes close at all
- icelancer 4y ago
  The level of denial people are willing to sink into regarding how good GPT-4 is compared to everything else is truly crazy. Not a single other project is an order of magnitude close to the quantitative and qualitative (actual experiential results, not just benchmarks) results that GPT-4 brings.
  - satvikpendem 4y ago
    I've had decent success with Open Assistant, an open source model. I'd say it's within the order of magnitude of ChatGPT, given the prompts I'm looking at, including reasoning prompts. This, I believe, is due to the overwhelmingly clean data that OA have managed to acquire through human volunteers.
  - spaceman_2020 4y ago
    I feel that there’s significant insecurity among a lot of coders about GPT-4. A lot of them are ignoring the pace of improvement and highlighting the few off chances where it gets things wrong.
    - AussieWog93 4y ago
      I've not used GPT-4, so it could be different, but regular old GPT-3.5 gets a _lot_ of things wrong.
      - snowe2010 4y ago
        GPT 4 is quite astounding. It might be wrong on occasion, but it will easily point you in the right direction most of the time. It still messes up, but like a twentieth of what 3.5 did. Honestly it is like an incredible rubber ducky for me. Not only can I just talk like I’m talking to a rubber duck but I can get fast, mostly informed, feedback that unblocks me. If I have a bunch of things competing for my attention I can ask gpt about one of them, a hard one, go do something else while it types out its answer, and then come back later and move on with that project.
        AussieWog93 4y ago
        Is a 95% reduction in errors an exaggeration or is it really that much better? Might just need to drop the $20/mo if it's really improved that much.
        revelio 4y ago
        I'm a recent convert, been experimenting with converting one PL to another. GPT-3.5 will get the gist of what the code is doing, and then provide what looks like a direct translation but differs in numerous details whilst having a bunch of other problems. GPT-4 does a correct translation, almost every time. It kills me that there's a waiting list for the API. I have put together some tools to integrate 3.5 into my workflow and it helps for my current task a lot (for others it's useless). But to really shine it needs to have API access to 4.
        IanCal 4y ago
        I recently finally got access to 4 in the API, it's good. It's much better imo at following the system prompt too. Faster than you see in chatgpt I think, not as fast as 3.5-turbo but definitely less tedious.
JumpCrisscross 4y ago
> How true is this? From playing around with Bard and Claude, GPT-4 seems to be significantly better I have at most moderate confidence in this hypothesis.

> other models are in the ballpark How true is this? From playing around with Bard and Claude, GPT-4 seems to be significantly better, especially around code generation / understanding.

letitgo12345 4y ago

People really don't seem to understand just how far ahead OAI is in its reasoning abilities (https://crfm.stanford.edu/helm/v0.2.2/?group=reasoning https://crfm.stanford.edu/helm/v0.2.2/?group=reasoning) Maybe PaLM is near there (it's not evaluated on that page) but nothing else even comes close at all

icelancer 4y ago
The level of denial people are willing to sink into regarding how good GPT-4 is compared to everything else is truly crazy. Not a single other project is an order of magnitude close to the quantitative and qualitative (actual experiential results, not just benchmarks) results that GPT-4 brings.
- satvikpendem 4y ago
  I've had decent success with Open Assistant, an open source model. I'd say it's within the order of magnitude of ChatGPT, given the prompts I'm looking at, including reasoning prompts. This, I believe, is due to the overwhelmingly clean data that OA have managed to acquire through human volunteers.
- spaceman_2020 4y ago
  I feel that there’s significant insecurity among a lot of coders about GPT-4. A lot of them are ignoring the pace of improvement and highlighting the few off chances where it gets things wrong.
  - AussieWog93 4y ago
    I've not used GPT-4, so it could be different, but regular old GPT-3.5 gets a _lot_ of things wrong.
    - snowe2010 4y ago
      GPT 4 is quite astounding. It might be wrong on occasion, but it will easily point you in the right direction most of the time. It still messes up, but like a twentieth of what 3.5 did. Honestly it is like an incredible rubber ducky for me. Not only can I just talk like I’m talking to a rubber duck but I can get fast, mostly informed, feedback that unblocks me. If I have a bunch of things competing for my attention I can ask gpt about one of them, a hard one, go do something else while it types out its answer, and then come back later and move on with that project.
      - AussieWog93 4y ago
        Is a 95% reduction in errors an exaggeration or is it really that much better? Might just need to drop the $20/mo if it's really improved that much.
        revelio 4y ago
        I'm a recent convert, been experimenting with converting one PL to another. GPT-3.5 will get the gist of what the code is doing, and then provide what looks like a direct translation but differs in numerous details whilst having a bunch of other problems. GPT-4 does a correct translation, almost every time. It kills me that there's a waiting list for the API. I have put together some tools to integrate 3.5 into my workflow and it helps for my current task a lot (for others it's useless). But to really shine it needs to have API access to 4.
        IanCal 4y ago
        I recently finally got access to 4 in the API, it's good. It's much better imo at following the system prompt too. Faster than you see in chatgpt I think, not as fast as 3.5-turbo but definitely less tedious.

JumpCrisscross 4y ago

> How true is this? From playing around with Bard and Claude, GPT-4 seems to be significantly better I have at most moderate confidence in this hypothesis.