5 ms·

The most interesting bit is that for the first time since the release of ChatGPT in December 2022, OpenAI does not have the lead on LLMs anymore. At least, for

The most interesting bit is that for the first time since the release of ChatGPT in December 2022, OpenAI does not have the lead on LLMs anymore. At least, for people who need large context windows, they would not be the first choice anymore.

refulgentis 3y ago
Claude’s very quietly better on everything but pricing, for a while, it just got buried because they announced on “AI Tuesday” (iirc gpt4 and Bing announcement day) The ChatGPT equivalent is 3x speed and was somewhere between ChatGPT and GPT4 on my TriviaQA benchmark replication I did Couple tweets with data and examples. Note they’re from 8 weeks ago, I know Claude got a version bump, GPT3.5/4 accessible via API seem the same. [1] brief and graphical summary of speed and TriviaQA https://twitter.com/jpohhhh/status/1638362982131351552?s=46&t=pxb6XO8RvMiZoZE1x8d58Q https://twitter.com/jpohhhh/status/1638362982131351552?s=46&... [2] ad hoc side by sides https://twitter.com/jpohhhh/status/1637316127314305024?s=46&t=pxb6XO8RvMiZoZE1x8d58Q https://twitter.com/jpohhhh/status/1637316127314305024?s=46&...
- com2kid 3y ago
  > I know Claude got a version bump, GPT3.5/4 accessible via API seem the same. GPT3.5 just got an update a few days ago that resulted in a pretty good improvement on its creativity. I saved some sample outputs from the previous March model, and for the same prompt the difference is quite dramatic. Prose is much less formulaic overall.
  - refulgentis 3y ago
    Thank you, every little comment I get from fellow boots on the ground is so valuable, lotta noise these days. Random Q, I don’t use the ChatGPT front end much past month or two, used it a week back and it seemed blazingly faster than my integration: do you have a sense of if it got faster too?
    - com2kid 3y ago
      Yeah I've noticed the front end for 3.5 responds almost instantly to complex questions. It'll pop out an entire page of code in under a couple seconds. I think API responses may be a bit faster than before, some of my queries that took 30s before now take around 20s, but they are obviously prioritizing their own site.
  - ndr_ 3y ago
    Is this update made visible somewhere? The language models offered on my Playground are still the ones from March, same with ChatGPT.
    - sanxiyn 3y ago
      On chat.openai.com below text box there is a link to ChatGPT release notes. Current link text is "ChatGPT May 3 Version". The link leads to https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-...
  - jiggawatts 3y ago
    Meanwhile GPT 4 got lobotomised around the same time. It can't even solve simplified versions of problems it had zero issues with just a week ago.
    - generalizations 3y ago
      Wait, really? I've only been using GPT4 and it seemed like it's been getting incrementally better. Do you have any test cases?
      - jiggawatts 3y ago
        Literally all of them! Comprehension, translation, problem solving, etc…. It’s worse at everything.
        refulgentis 3y ago
        Nah
        generalizations 3y ago
        You know, I'm noticing it seems to work better on off-hours - I get better quality responses late at night. I wonder if they're using a lighter model when there's more demand on the system.
- ilaksh 3y ago
  How is the code generation of Claude?
  - esafak 3y ago
    And is code generation ability equivalent to code understanding and search ability?
  - technics256 3y ago
    I have access to claude. It's not bad, but decently behind gpt4 for code
  - refulgentis 3y ago
    Note, all impressions based on Claude 1.2, got an email from Poe in the last week saying it was version bumped to 1.3 with a focus on coding improvements. Impressions: Bad enough compared to GPT-4 that I default to GPT-4. I think if I had api access I’d use it instead, right now it requires more coaxing, and using Poe. I did find “long-term” chats went better, was really impressed with how it held up when I was asking it a nasty problem that was hard to even communicate verbally. Wrong at first, but as I conversed it was a real conversation. GPT4 seems to circle a lower optima. My academic guess it’s what Anthropic calls it “sycophancy” in its papers, tldr GPT really really wants to do more like what’s in the context, so the longer the conversation with initial errors goes, it’s actually harder to talk it out of the errors.
    - laratied 3y ago
      [dead]
sebzim4500 3y ago
GPT-4 still leads in the chatbot arena[1] but at least it is a two horse race now. [1] https://lmsys.org/blog/2023-05-10-leaderboard/ https://lmsys.org/blog/2023-05-10-leaderboard/
speedylight 3y ago
Well Google decided to stop releasing their AI research.

refulgentis 3y ago

Claude’s very quietly better on everything but pricing, for a while, it just got buried because they announced on “AI Tuesday” (iirc gpt4 and Bing announcement day) The ChatGPT equivalent is 3x speed and was somewhere between ChatGPT and GPT4 on my TriviaQA benchmark replication I did Couple tweets with data and examples. Note they’re from 8 weeks ago, I know Claude got a version bump, GPT3.5/4 accessible via API seem the same. [1] brief and graphical summary of speed and TriviaQA https://twitter.com/jpohhhh/status/1638362982131351552?s=46&t=pxb6XO8RvMiZoZE1x8d58Q https://twitter.com/jpohhhh/status/1638362982131351552?s=46&... [2] ad hoc side by sides https://twitter.com/jpohhhh/status/1637316127314305024?s=46&t=pxb6XO8RvMiZoZE1x8d58Q https://twitter.com/jpohhhh/status/1637316127314305024?s=46&...

com2kid 3y ago
> I know Claude got a version bump, GPT3.5/4 accessible via API seem the same. GPT3.5 just got an update a few days ago that resulted in a pretty good improvement on its creativity. I saved some sample outputs from the previous March model, and for the same prompt the difference is quite dramatic. Prose is much less formulaic overall.
- refulgentis 3y ago
  Thank you, every little comment I get from fellow boots on the ground is so valuable, lotta noise these days. Random Q, I don’t use the ChatGPT front end much past month or two, used it a week back and it seemed blazingly faster than my integration: do you have a sense of if it got faster too?
  - com2kid 3y ago
    Yeah I've noticed the front end for 3.5 responds almost instantly to complex questions. It'll pop out an entire page of code in under a couple seconds. I think API responses may be a bit faster than before, some of my queries that took 30s before now take around 20s, but they are obviously prioritizing their own site.
- ndr_ 3y ago
  Is this update made visible somewhere? The language models offered on my Playground are still the ones from March, same with ChatGPT.
  - sanxiyn 3y ago
    On chat.openai.com below text box there is a link to ChatGPT release notes. Current link text is "ChatGPT May 3 Version". The link leads to https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-...
- jiggawatts 3y ago
  Meanwhile GPT 4 got lobotomised around the same time. It can't even solve simplified versions of problems it had zero issues with just a week ago.
  - generalizations 3y ago
    Wait, really? I've only been using GPT4 and it seemed like it's been getting incrementally better. Do you have any test cases?
    - jiggawatts 3y ago
      Literally all of them! Comprehension, translation, problem solving, etc…. It’s worse at everything.
      - refulgentis 3y ago
        Nah
      - generalizations 3y ago
        You know, I'm noticing it seems to work better on off-hours - I get better quality responses late at night. I wonder if they're using a lighter model when there's more demand on the system.
ilaksh 3y ago
How is the code generation of Claude?
- esafak 3y ago
  And is code generation ability equivalent to code understanding and search ability?
- technics256 3y ago
  I have access to claude. It's not bad, but decently behind gpt4 for code
- refulgentis 3y ago
  Note, all impressions based on Claude 1.2, got an email from Poe in the last week saying it was version bumped to 1.3 with a focus on coding improvements. Impressions: Bad enough compared to GPT-4 that I default to GPT-4. I think if I had api access I’d use it instead, right now it requires more coaxing, and using Poe. I did find “long-term” chats went better, was really impressed with how it held up when I was asking it a nasty problem that was hard to even communicate verbally. Wrong at first, but as I conversed it was a real conversation. GPT4 seems to circle a lower optima. My academic guess it’s what Anthropic calls it “sycophancy” in its papers, tldr GPT really really wants to do more like what’s in the context, so the longer the conversation with initial errors goes, it’s actually harder to talk it out of the errors.
  - laratied 3y ago
    [dead]

sebzim4500 3y ago

GPT-4 still leads in the chatbot arena[1] but at least it is a two horse race now. [1] https://lmsys.org/blog/2023-05-10-leaderboard/ https://lmsys.org/blog/2023-05-10-leaderboard/

speedylight 3y ago

Well Google decided to stop releasing their AI research.