5 ms·

I think this post makes a few good points, certainly the parabolic trajectory Scale seems to be on is at least suggestive if not conclusive that there’s a lot m

I think this post makes a few good points, certainly the parabolic trajectory Scale seems to be on is at least suggestive if not conclusive that there’s a lot more going on now than just big text crawls. And Phi-3 is something else, even from relatively limited time playing with it, so that’s useful signal for anyone who hadn’t looked at it yet. Wildly cool stuff. It seems weird to not mention Anthropic or Mistral or FAIR pretty much at all: they’re all pretty clearly on more modern architectures at least as concerns capability per weight and Instruct-style stuff. I’m part of a now nontrivial group who regards Opus as basically shattering GPT-4-{0125, 1106}-Preview (which is basically the same as 4o for pure language modalities) on basically everything I care about, and LLaMA3 is just about there as well, maybe not quite Opus, comparable if you ignore trivially gamed metrics like MMLU. And I have no idea why we’re talking about GPT-5 when there’s little if any verifiable evidence it even exists as a training run tracking to completion. Maybe it is, maybe not, but let’s get a look at it rather than just assume that it’s going to lap the labs that are currently pushing the pace now?

ein0p 2y ago
Have you actually tried using Opus side by side with GPT4 on “work” related stuff? GPT4 is way better in my experience, to the point where I cancelled my Opus subscription after just a couple of months.
- deleted 2y ago
  [deleted]
- benreesman 2y ago
  I make an effort to use both and several high-capability open tunes every day (it’s not literally every day but I have keyboard shortcuts for all of them). Opus historically had issues with minor typographical errors, though recently that seems to not happen often, lots of very sharp people at Anthropic. So a month ago if I wanted something from Opus I’d run it through a cleanup pass courtesy of one of the other ones, but even my old standby dolphin-8x7 can clean up typos. 1106 can as well, but all else equal I don’t want to be sending my stuff to any black box data warehouse and I’m always surprised so many other sophisticated people don’t share the preference. My personal eyeball capability check is to posit a gauge symmetry and ask what it thinks the implied conserved quantity is, and I’ve yet to see Opus not crush that relative to anything else, including real footnotes. On coding I usually hand it a Prisma schema and ask for a proto3/gRPC definition that is a good way to interact with it, Opus in my personal experience also dominates there. If you have an example of a task that represents a counter example I’d be grateful for another little integration test for my personal ad-hoc model card. I want to know the best tool for every job.
  - refulgentis 2y ago
    Opus hasn't changed. I really don't like how condescending your root comment is, I don't have any idea how all these extra things you're disappointed in are relevant at all to the actual topic.
- virgildotcodes 2y ago
  I feel like Opus is much better at creative writing in terms of sounding more natural and less formulaic, but GPT4 does beat it on just about everything else.
  - benreesman 2y ago
    I replied to a sibling with a few of my ad-hoc tasks. I’d likewise be grateful if you’d lend me an example or two that should be on my little ad-hoc task set.
    - CSMastermind 2y ago
      Which direction are you looking for? Ones where GPT-4o does better or ones where Opus does better? Assuming the former, here's an example I had yesterday where Opus was never able to formulate a correct response even with multiple follow-ups but GPT-4o got it with only two follow ups: > I have a postgres database with a table called projects. On that table is a column called designs that is of json type and is nullable as well as a column called id that is of type text. The data in the designs column will be an array of objects (assuming it's not null). Each object in the array will have a field called price that will be a string. Most of these prices will be in the format $#,### where # is a number. However some will be in the format # ### US$. This second format where the numbers are separated by spaces instead of commas and the dollar sign appears at the end of the string instead of the beginning with an extraneous US is incorrect. What SQL can I run to migrate all prices that are incorrectly formatted to the correct format? Alternatively, this is a non-tech related one that GPT-4 gets correct that no other model does: > In 388 BC, the boxer Eupolus of Thessaly defeated three opponents at the ancient Olympics. Funded by the men involved, several bronze statues of Zeus were erected, with inscriptions detailing the events. Why did the four men prefer not to have the statues around? I like that one because Opus will tell you that you're wrong and mistaken, which is pretty funny while GPT-4 answer correctly with the actual context.
      - ppsreejith 2y ago
        Very interesting prompt regarding the boxer Eupolus of Thessaly. Gemini Advanced gets this wrong as well as Llama 3 70b (as run on groq.com). However, if I start with: "What is the earliest record of cheating in the olympic games?" Then all models get the question right. It's surprising that GPT-4 gets it right on the first go.
- Zambyte 2y ago
  That's really interesting to me. I've been using both through my Kagi subscription, but I always find myself favoring the quality of Opus. I generally use GPT 4o if I don't want to wait for a slow response from Opus, and I use Opus when I want the highest quality.
  - benreesman 2y ago
    It’s likewise my experience that on e.g. TTFT is higher on Anthropic’s API. But it’s not by a ton, and it could be just less hyper-scale network infrastructure. I don’t recall what Anthropic has raised but IIRC it wasn’t “peer with everyone” levels.
    - seanhunter 2y ago
      I think they've raised just over $5B, consisting of $1B from everyone other than amazon (750m at [1] and 450m at [2]) and then $4B from amazon[3]. The reason I seperate them is I think a lot of the $4B from amazon is in the form of AWS credits rather than cash. [1] https://www.bloomberg.com/news/articles/2023-12-21/anthropic-in-talks-to-raise-750-million-led-by-menlo-ventures https://www.bloomberg.com/news/articles/2023-12-21/anthropic... [2] https://www.anthropic.com/news/anthropic-series-c https://www.anthropic.com/news/anthropic-series-c [3] https://www.aboutamazon.com/news/company-news/amazon-anthropic-ai-investment https://www.aboutamazon.com/news/company-news/amazon-anthrop...
akira2501 2y ago
> And Phi-3 is something else, It's great that we get to keep saying this. I wonder if that's because we have no objective statistics to measure these projects by.
- benreesman 2y ago
  IMHO it’s easier than people largely seem to imply. Make it easy to try everything, let people decide for themselves what works best for them. Ya know, like a market.
  - akira2501 2y ago
    No, that's "marketing." An actual market requires informed consumers and standardized metrics between competitors. They also typically include publications about the market positioned towards the consumer, almost like ya know, Consumer Reports.
dannyw 2y ago
Phi3 was trained on benchmarks, it’s contaminated and deceitful. Actual performance is much worse in my experience.
- lostmsu 2y ago
  Phi-3-Mini has the same ELO on chatbot arena as the oldest GPT-3.5-Turbo. It is an 8GB model (~4B paramters?)
fnordpiglet 2y ago
OpenAI has publicly said they’ve started building GPT5. https://openai.com/index/openai-board-forms-safety-and-security-committee/ https://openai.com/index/openai-board-forms-safety-and-secur...

ein0p 2y ago

Have you actually tried using Opus side by side with GPT4 on “work” related stuff? GPT4 is way better in my experience, to the point where I cancelled my Opus subscription after just a couple of months.

deleted 2y ago
[deleted]
benreesman 2y ago
I make an effort to use both and several high-capability open tunes every day (it’s not literally every day but I have keyboard shortcuts for all of them). Opus historically had issues with minor typographical errors, though recently that seems to not happen often, lots of very sharp people at Anthropic. So a month ago if I wanted something from Opus I’d run it through a cleanup pass courtesy of one of the other ones, but even my old standby dolphin-8x7 can clean up typos. 1106 can as well, but all else equal I don’t want to be sending my stuff to any black box data warehouse and I’m always surprised so many other sophisticated people don’t share the preference. My personal eyeball capability check is to posit a gauge symmetry and ask what it thinks the implied conserved quantity is, and I’ve yet to see Opus not crush that relative to anything else, including real footnotes. On coding I usually hand it a Prisma schema and ask for a proto3/gRPC definition that is a good way to interact with it, Opus in my personal experience also dominates there. If you have an example of a task that represents a counter example I’d be grateful for another little integration test for my personal ad-hoc model card. I want to know the best tool for every job.
- refulgentis 2y ago
  Opus hasn't changed. I really don't like how condescending your root comment is, I don't have any idea how all these extra things you're disappointed in are relevant at all to the actual topic.
virgildotcodes 2y ago
I feel like Opus is much better at creative writing in terms of sounding more natural and less formulaic, but GPT4 does beat it on just about everything else.
- benreesman 2y ago
  I replied to a sibling with a few of my ad-hoc tasks. I’d likewise be grateful if you’d lend me an example or two that should be on my little ad-hoc task set.
  - CSMastermind 2y ago
    Which direction are you looking for? Ones where GPT-4o does better or ones where Opus does better? Assuming the former, here's an example I had yesterday where Opus was never able to formulate a correct response even with multiple follow-ups but GPT-4o got it with only two follow ups: > I have a postgres database with a table called projects. On that table is a column called designs that is of json type and is nullable as well as a column called id that is of type text. The data in the designs column will be an array of objects (assuming it's not null). Each object in the array will have a field called price that will be a string. Most of these prices will be in the format $#,### where # is a number. However some will be in the format # ### US$. This second format where the numbers are separated by spaces instead of commas and the dollar sign appears at the end of the string instead of the beginning with an extraneous US is incorrect. What SQL can I run to migrate all prices that are incorrectly formatted to the correct format? Alternatively, this is a non-tech related one that GPT-4 gets correct that no other model does: > In 388 BC, the boxer Eupolus of Thessaly defeated three opponents at the ancient Olympics. Funded by the men involved, several bronze statues of Zeus were erected, with inscriptions detailing the events. Why did the four men prefer not to have the statues around? I like that one because Opus will tell you that you're wrong and mistaken, which is pretty funny while GPT-4 answer correctly with the actual context.
    - ppsreejith 2y ago
      Very interesting prompt regarding the boxer Eupolus of Thessaly. Gemini Advanced gets this wrong as well as Llama 3 70b (as run on groq.com). However, if I start with: "What is the earliest record of cheating in the olympic games?" Then all models get the question right. It's surprising that GPT-4 gets it right on the first go.
Zambyte 2y ago
That's really interesting to me. I've been using both through my Kagi subscription, but I always find myself favoring the quality of Opus. I generally use GPT 4o if I don't want to wait for a slow response from Opus, and I use Opus when I want the highest quality.
- benreesman 2y ago
  It’s likewise my experience that on e.g. TTFT is higher on Anthropic’s API. But it’s not by a ton, and it could be just less hyper-scale network infrastructure. I don’t recall what Anthropic has raised but IIRC it wasn’t “peer with everyone” levels.
  - seanhunter 2y ago
    I think they've raised just over $5B, consisting of $1B from everyone other than amazon (750m at [1] and 450m at [2]) and then $4B from amazon[3]. The reason I seperate them is I think a lot of the $4B from amazon is in the form of AWS credits rather than cash. [1] https://www.bloomberg.com/news/articles/2023-12-21/anthropic-in-talks-to-raise-750-million-led-by-menlo-ventures https://www.bloomberg.com/news/articles/2023-12-21/anthropic... [2] https://www.anthropic.com/news/anthropic-series-c https://www.anthropic.com/news/anthropic-series-c [3] https://www.aboutamazon.com/news/company-news/amazon-anthropic-ai-investment https://www.aboutamazon.com/news/company-news/amazon-anthrop...

akira2501 2y ago

> And Phi-3 is something else, It's great that we get to keep saying this. I wonder if that's because we have no objective statistics to measure these projects by.

benreesman 2y ago
IMHO it’s easier than people largely seem to imply. Make it easy to try everything, let people decide for themselves what works best for them. Ya know, like a market.
- akira2501 2y ago
  No, that's "marketing." An actual market requires informed consumers and standardized metrics between competitors. They also typically include publications about the market positioned towards the consumer, almost like ya know, Consumer Reports.

dannyw 2y ago

Phi3 was trained on benchmarks, it’s contaminated and deceitful. Actual performance is much worse in my experience.

lostmsu 2y ago
Phi-3-Mini has the same ELO on chatbot arena as the oldest GPT-3.5-Turbo. It is an 8GB model (~4B paramters?)

fnordpiglet 2y ago

OpenAI has publicly said they’ve started building GPT5. https://openai.com/index/openai-board-forms-safety-and-security-committee/ https://openai.com/index/openai-board-forms-safety-and-secur...