7 ms·

Claude’s very quietly better on everything but pricing, for a while, it just got buried because they announced on “AI Tuesday” (iirc gpt4 and Bing announcement

Claude’s very quietly better on everything but pricing, for a while, it just got buried because they announced on “AI Tuesday” (iirc gpt4 and Bing announcement day) The ChatGPT equivalent is 3x speed and was somewhere between ChatGPT and GPT4 on my TriviaQA benchmark replication I did Couple tweets with data and examples. Note they’re from 8 weeks ago, I know Claude got a version bump, GPT3.5/4 accessible via API seem the same. [1] brief and graphical summary of speed and TriviaQA https://twitter.com/jpohhhh/status/1638362982131351552?s=46&t=pxb6XO8RvMiZoZE1x8d58Q https://twitter.com/jpohhhh/status/1638362982131351552?s=46&... [2] ad hoc side by sides https://twitter.com/jpohhhh/status/1637316127314305024?s=46&t=pxb6XO8RvMiZoZE1x8d58Q https://twitter.com/jpohhhh/status/1637316127314305024?s=46&...

com2kid 4y ago
> I know Claude got a version bump, GPT3.5/4 accessible via API seem the same. GPT3.5 just got an update a few days ago that resulted in a pretty good improvement on its creativity. I saved some sample outputs from the previous March model, and for the same prompt the difference is quite dramatic. Prose is much less formulaic overall.
- refulgentis 4y ago
  Thank you, every little comment I get from fellow boots on the ground is so valuable, lotta noise these days. Random Q, I don’t use the ChatGPT front end much past month or two, used it a week back and it seemed blazingly faster than my integration: do you have a sense of if it got faster too?
  - com2kid 4y ago
    Yeah I've noticed the front end for 3.5 responds almost instantly to complex questions. It'll pop out an entire page of code in under a couple seconds. I think API responses may be a bit faster than before, some of my queries that took 30s before now take around 20s, but they are obviously prioritizing their own site.
- ndr_ 4y ago
  Is this update made visible somewhere? The language models offered on my Playground are still the ones from March, same with ChatGPT.
  - sanxiyn 4y ago
    On chat.openai.com below text box there is a link to ChatGPT release notes. Current link text is "ChatGPT May 3 Version". The link leads to https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-...
- jiggawatts 4y ago
  Meanwhile GPT 4 got lobotomised around the same time. It can't even solve simplified versions of problems it had zero issues with just a week ago.
  - generalizations 4y ago
    Wait, really? I've only been using GPT4 and it seemed like it's been getting incrementally better. Do you have any test cases?
    - jiggawatts 4y ago
      Literally all of them! Comprehension, translation, problem solving, etc…. It’s worse at everything.
      - refulgentis 4y ago
        Nah
      - generalizations 4y ago
        You know, I'm noticing it seems to work better on off-hours - I get better quality responses late at night. I wonder if they're using a lighter model when there's more demand on the system.
ilaksh 4y ago
How is the code generation of Claude?
- esafak 4y ago
  And is code generation ability equivalent to code understanding and search ability?
- technics256 4y ago
  I have access to claude. It's not bad, but decently behind gpt4 for code
- refulgentis 4y ago
  Note, all impressions based on Claude 1.2, got an email from Poe in the last week saying it was version bumped to 1.3 with a focus on coding improvements. Impressions: Bad enough compared to GPT-4 that I default to GPT-4. I think if I had api access I’d use it instead, right now it requires more coaxing, and using Poe. I did find “long-term” chats went better, was really impressed with how it held up when I was asking it a nasty problem that was hard to even communicate verbally. Wrong at first, but as I conversed it was a real conversation. GPT4 seems to circle a lower optima. My academic guess it’s what Anthropic calls it “sycophancy” in its papers, tldr GPT really really wants to do more like what’s in the context, so the longer the conversation with initial errors goes, it’s actually harder to talk it out of the errors.
  - laratied 4y ago
    [dead]

com2kid 4y ago

> I know Claude got a version bump, GPT3.5/4 accessible via API seem the same. GPT3.5 just got an update a few days ago that resulted in a pretty good improvement on its creativity. I saved some sample outputs from the previous March model, and for the same prompt the difference is quite dramatic. Prose is much less formulaic overall.

refulgentis 4y ago
Thank you, every little comment I get from fellow boots on the ground is so valuable, lotta noise these days. Random Q, I don’t use the ChatGPT front end much past month or two, used it a week back and it seemed blazingly faster than my integration: do you have a sense of if it got faster too?
- com2kid 4y ago
  Yeah I've noticed the front end for 3.5 responds almost instantly to complex questions. It'll pop out an entire page of code in under a couple seconds. I think API responses may be a bit faster than before, some of my queries that took 30s before now take around 20s, but they are obviously prioritizing their own site.
ndr_ 4y ago
Is this update made visible somewhere? The language models offered on my Playground are still the ones from March, same with ChatGPT.
- sanxiyn 4y ago
  On chat.openai.com below text box there is a link to ChatGPT release notes. Current link text is "ChatGPT May 3 Version". The link leads to https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-...
jiggawatts 4y ago
Meanwhile GPT 4 got lobotomised around the same time. It can't even solve simplified versions of problems it had zero issues with just a week ago.
- generalizations 4y ago
  Wait, really? I've only been using GPT4 and it seemed like it's been getting incrementally better. Do you have any test cases?
  - jiggawatts 4y ago
    Literally all of them! Comprehension, translation, problem solving, etc…. It’s worse at everything.
    - refulgentis 4y ago
      Nah
    - generalizations 4y ago
      You know, I'm noticing it seems to work better on off-hours - I get better quality responses late at night. I wonder if they're using a lighter model when there's more demand on the system.

ilaksh 4y ago

How is the code generation of Claude?

esafak 4y ago
And is code generation ability equivalent to code understanding and search ability?
technics256 4y ago
I have access to claude. It's not bad, but decently behind gpt4 for code
refulgentis 4y ago
Note, all impressions based on Claude 1.2, got an email from Poe in the last week saying it was version bumped to 1.3 with a focus on coding improvements. Impressions: Bad enough compared to GPT-4 that I default to GPT-4. I think if I had api access I’d use it instead, right now it requires more coaxing, and using Poe. I did find “long-term” chats went better, was really impressed with how it held up when I was asking it a nasty problem that was hard to even communicate verbally. Wrong at first, but as I conversed it was a real conversation. GPT4 seems to circle a lower optima. My academic guess it’s what Anthropic calls it “sycophancy” in its papers, tldr GPT really really wants to do more like what’s in the context, so the longer the conversation with initial errors goes, it’s actually harder to talk it out of the errors.
- laratied 4y ago
  [dead]