5 ms·

Surpassing GPT4 is huge for any model, very impressive to pull off. But then again...GPT4 is a year old and OpenAI has not yet revealed their next-gen model.

HarHarVeryFunny 3y ago
Sure, OpenAI's next model would be expected to regain the lead, just due to their head start, but this level of catch-up from Anthropic is extremely impressive. Bear in mind that GPT-3 was published ("Language Models are Few-Shot Learners") in 2020, and Anthropic were only founded after that in 2021. So, with OpenAI having three generations under their belt, Anthropic came from nothing (at least in terms of models - of course some team members had the know-how of being ex. OpenAI) and are, temporarily at least, now ahead of OpenAI in some of these benchmarks. I'd assume that OpenAI's next-gen model (GPT-5 or whatever they will choose to call it) has already finished training and is now being fine tuned and evaluated for safety, but Anthropic's cause d'etre is safety and I doubt they have skimped on this to rush this model out.
- aaomidi 3y ago
  Anthropic is also not really a traditional startup. It’s just some large companies in a trench coat.
  - hobofan 3y ago
    How so? Because they have taken large investments from Amazon and Google? Or would you also characterize OpenAI as "Microsoft in a trench coat"?
    - pavlov 3y ago
      > 'would you also characterize OpenAI as "Microsoft in a trench coat"?' Elon Musk seems to think that, based on his recent lawsuit. I wouldn't agree but the argument has some validity if you look at the role Microsoft played in reversing the Altman firing.
    - bugglebeetle 3y ago
      100% OpenAI is Microsoft in a trenchcoat.
      - HarHarVeryFunny 3y ago
        They are funded mostly by Microsoft, and dependent on them for compute (which is what this funding is mostly buying), but I'd hardly characterize that as meaning they are "Microsoft in a trenchcoat". It's not normal to identify startups as being their "VC in a trenchcoat", even if they are dependent on the money for growth.
        bugglebeetle 3y ago
        Satya Nadella during the OpenAI leadership fiasco: “We have all of the rights to continue the innovation, not just to serve the product, but we can, you know, go and just do what we were doing in partnership ourselves. And so we have the people, we have the compute, we have the data, we have everything.” Doesn’t sound like a startup-investor relationship to me!
        HarHarVeryFunny 3y ago
        Sure, but that's just saying that Microsoft as investor has some rights to the underlying tech. There are limits to this though, which we may fairly soon be nearing. I believe the agreement says that Microsoft's rights to the tech (model + weights? training data? -- not sure how specific it is) end once AGI is achieved, however that is evaluated. But again, this is not to say that OpenAI is "Microsoft in a trenchcoat". Microsoft don't have developers at OpenAI, weren't behind the tech in any way, etc. Their $10B investment bought them some short-term insurance in limited rights to the tech. It is what is is.
        bugglebeetle 3y ago
        “We have everything” is not “some underlying rights to the tech.” I dunno what the angle is on minimizing here, but I’ll take the head of Microsoft at his word vs. more strained explanations about why this isn’t the case.
    - aaomidi 3y ago
      Absolutely to OpenAI being Microsoft in a trench coat. This is not an uncommon tactic for companies to use.
- appplication 3y ago
  What this really says to me is the indefensibility of any current advances. There’s really cool stuff going on right now, but anyone can do it. Not to say anyone can push the limits of research, but once the cat’s out of the bag, anyone with a few $B and dozen engineers can replicate a model that’s indistinguishably good from best in class to most users.
  - zurfer 3y ago
    Barrier to entry with "few $B" is pretty high. Especially since the scaling laws indicate that it's only getting more expensive. And even if you manage to raise $Bs, you still need to be clever on how to deploy it (talent, compute, data) ...
    - appplication 3y ago
      You’re totally right, a few $B is not something any of us are bootstrapping. But there is no secret sauce (at least none that stays secret for long), no meaningful patents, no network/platform effect, and virtually no ability to lock in customers. Compare to other traditional tech companies… think Uber/AirBnB/Databricks/etc. Their product isn’t an algorithm that a competitor can spin up in 6 months. These companies create real moats, for better or worse, which significantly reduce the ability for competitors to enter, even with tranches of cash. In contrast, essentially every product we’ve seen in the AI space is very replicable, and any differentiation is largely marginal, under the hood, and the details of which are obscured from customers.
      - zurfer 3y ago
        Every big tech in the beginning looked fragile/no moats. I think we'll see that data, knowledge and intelligence compound and at some point it will be as hard to penetrate as Meta's network effects.
        HarHarVeryFunny 3y ago
        Maybe consolidate as well as compound. There's a tendency for any mature industry (which may initially have been bustling with competitors) to eventually consolidate into three players, and while we're currently at the point where it seems a well-funded new entrant can catch up with the leaders, that will likely become much harder in the future as tech advances. Never say never though - look at Tesla coming out of nowhere to push the big three automakers around! Eventually the established players become too complacent and set in their ways, creating an opening for a smaller more nimble competitor with a better idea. I don't think LLMs are the ultimate form of AI/AGI though. Eventually we'll figure out a better brain-inspired approach that learns continually from it's own experimentation and experience. Perhaps this change of approach will be when some much smaller competitor (someone like John Carmack, perhaps) rapidly come from nowhere and catch the big three flat footed as they tend to their ginormous LLM training sets, infrastructure and entrenched products.
- lr1970 3y ago
  > Bear in mind that GPT-3 was published ("Language Models are Few-Shot Learners") in 2020, and Anthropic were only founded after that in 2021. Keep in mind that Antropic was founded by former OpenAI people (Dario Amadei and others). Both companies share a lot of R&D "DNA".
bugglebeetle 3y ago
MMLU is pretty much the only stat on there that matters, as it correlates to multitask reasoning ability. Here, they outpace GPT-4 by a smidge, but even that is impressive because I don’t think anyone else’s has to date.
- hackerlight 3y ago
  How can they avoid the contents from leaking into the training set somewhere in their internet scrape?
- jasonjmcghee 3y ago
  I still don't trust benchmarks, but they've come a long way. It's genuinely outperforming GPT4 in my manual tests.
- rafaelero 3y ago
  MMLU is garbage. A lot of incorrect answers there.
  - bugglebeetle 3y ago
    And yet it’s still a good indicator of general performance. Any model that scores under GPT-4 on that benchmark, but above it in other, tends to be worse overall.
imjonse 3y ago
From the blog's footnote: "In addition, we’d like to note that engineers have worked to optimize prompts and few-shot samples for evaluations and reported higher scores for a newer GPT-4T model"
thefourthchime 3y ago
ChatGPT4 gets updated all the time, the latest are: GPT-4-1106-preview GPT-4-0125-preview See: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
VirusNewbie 3y ago
Right but the people who were instrumental in the creation of GPT are now...working at Anthropic.

HarHarVeryFunny 3y ago

Sure, OpenAI's next model would be expected to regain the lead, just due to their head start, but this level of catch-up from Anthropic is extremely impressive. Bear in mind that GPT-3 was published ("Language Models are Few-Shot Learners") in 2020, and Anthropic were only founded after that in 2021. So, with OpenAI having three generations under their belt, Anthropic came from nothing (at least in terms of models - of course some team members had the know-how of being ex. OpenAI) and are, temporarily at least, now ahead of OpenAI in some of these benchmarks. I'd assume that OpenAI's next-gen model (GPT-5 or whatever they will choose to call it) has already finished training and is now being fine tuned and evaluated for safety, but Anthropic's cause d'etre is safety and I doubt they have skimped on this to rush this model out.

aaomidi 3y ago
Anthropic is also not really a traditional startup. It’s just some large companies in a trench coat.
- hobofan 3y ago
  How so? Because they have taken large investments from Amazon and Google? Or would you also characterize OpenAI as "Microsoft in a trench coat"?
  - pavlov 3y ago
    > 'would you also characterize OpenAI as "Microsoft in a trench coat"?' Elon Musk seems to think that, based on his recent lawsuit. I wouldn't agree but the argument has some validity if you look at the role Microsoft played in reversing the Altman firing.
  - bugglebeetle 3y ago
    100% OpenAI is Microsoft in a trenchcoat.
    - HarHarVeryFunny 3y ago
      They are funded mostly by Microsoft, and dependent on them for compute (which is what this funding is mostly buying), but I'd hardly characterize that as meaning they are "Microsoft in a trenchcoat". It's not normal to identify startups as being their "VC in a trenchcoat", even if they are dependent on the money for growth.
      - bugglebeetle 3y ago
        Satya Nadella during the OpenAI leadership fiasco: “We have all of the rights to continue the innovation, not just to serve the product, but we can, you know, go and just do what we were doing in partnership ourselves. And so we have the people, we have the compute, we have the data, we have everything.” Doesn’t sound like a startup-investor relationship to me!
        HarHarVeryFunny 3y ago
        Sure, but that's just saying that Microsoft as investor has some rights to the underlying tech. There are limits to this though, which we may fairly soon be nearing. I believe the agreement says that Microsoft's rights to the tech (model + weights? training data? -- not sure how specific it is) end once AGI is achieved, however that is evaluated. But again, this is not to say that OpenAI is "Microsoft in a trenchcoat". Microsoft don't have developers at OpenAI, weren't behind the tech in any way, etc. Their $10B investment bought them some short-term insurance in limited rights to the tech. It is what is is.
        bugglebeetle 3y ago
        “We have everything” is not “some underlying rights to the tech.” I dunno what the angle is on minimizing here, but I’ll take the head of Microsoft at his word vs. more strained explanations about why this isn’t the case.
  - aaomidi 3y ago
    Absolutely to OpenAI being Microsoft in a trench coat. This is not an uncommon tactic for companies to use.
appplication 3y ago
What this really says to me is the indefensibility of any current advances. There’s really cool stuff going on right now, but anyone can do it. Not to say anyone can push the limits of research, but once the cat’s out of the bag, anyone with a few $B and dozen engineers can replicate a model that’s indistinguishably good from best in class to most users.
- zurfer 3y ago
  Barrier to entry with "few $B" is pretty high. Especially since the scaling laws indicate that it's only getting more expensive. And even if you manage to raise $Bs, you still need to be clever on how to deploy it (talent, compute, data) ...
  - appplication 3y ago
    You’re totally right, a few $B is not something any of us are bootstrapping. But there is no secret sauce (at least none that stays secret for long), no meaningful patents, no network/platform effect, and virtually no ability to lock in customers. Compare to other traditional tech companies… think Uber/AirBnB/Databricks/etc. Their product isn’t an algorithm that a competitor can spin up in 6 months. These companies create real moats, for better or worse, which significantly reduce the ability for competitors to enter, even with tranches of cash. In contrast, essentially every product we’ve seen in the AI space is very replicable, and any differentiation is largely marginal, under the hood, and the details of which are obscured from customers.
    - zurfer 3y ago
      Every big tech in the beginning looked fragile/no moats. I think we'll see that data, knowledge and intelligence compound and at some point it will be as hard to penetrate as Meta's network effects.
      - HarHarVeryFunny 3y ago
        Maybe consolidate as well as compound. There's a tendency for any mature industry (which may initially have been bustling with competitors) to eventually consolidate into three players, and while we're currently at the point where it seems a well-funded new entrant can catch up with the leaders, that will likely become much harder in the future as tech advances. Never say never though - look at Tesla coming out of nowhere to push the big three automakers around! Eventually the established players become too complacent and set in their ways, creating an opening for a smaller more nimble competitor with a better idea. I don't think LLMs are the ultimate form of AI/AGI though. Eventually we'll figure out a better brain-inspired approach that learns continually from it's own experimentation and experience. Perhaps this change of approach will be when some much smaller competitor (someone like John Carmack, perhaps) rapidly come from nowhere and catch the big three flat footed as they tend to their ginormous LLM training sets, infrastructure and entrenched products.
lr1970 3y ago
> Bear in mind that GPT-3 was published ("Language Models are Few-Shot Learners") in 2020, and Anthropic were only founded after that in 2021. Keep in mind that Antropic was founded by former OpenAI people (Dario Amadei and others). Both companies share a lot of R&D "DNA".

bugglebeetle 3y ago

MMLU is pretty much the only stat on there that matters, as it correlates to multitask reasoning ability. Here, they outpace GPT-4 by a smidge, but even that is impressive because I don’t think anyone else’s has to date.

hackerlight 3y ago
How can they avoid the contents from leaking into the training set somewhere in their internet scrape?
jasonjmcghee 3y ago
I still don't trust benchmarks, but they've come a long way. It's genuinely outperforming GPT4 in my manual tests.
rafaelero 3y ago
MMLU is garbage. A lot of incorrect answers there.
- bugglebeetle 3y ago
  And yet it’s still a good indicator of general performance. Any model that scores under GPT-4 on that benchmark, but above it in other, tends to be worse overall.

imjonse 3y ago

From the blog's footnote: "In addition, we’d like to note that engineers have worked to optimize prompts and few-shot samples for evaluations and reported higher scores for a newer GPT-4T model"

thefourthchime 3y ago

ChatGPT4 gets updated all the time, the latest are: GPT-4-1106-preview GPT-4-0125-preview See: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

VirusNewbie 3y ago

Right but the people who were instrumental in the creation of GPT are now...working at Anthropic.