3 ms·

> If I had to guess the pro and flash variants are 5x to 10x smaller than opus and gpt-5 class models. I really doubt it, especially Pro. If anything I wouldn'

by deaux 6mo ago

> If I had to guess the pro and flash variants are 5x to 10x smaller than opus and gpt-5 class models. I really doubt it, especially Pro. If anything I wouldn't be surprised if their hardware lets them run bigger models more cheaply and quickly than the others. Pro is probably smaller than GPT 5.4 and Opus 4.6 (looks like 4.7 decreased in size), but 5x seems way too much. IMO Gemini 3 Pro is the most "intelligent" in an all-round human way. Especially in the humanities. It's highly knowledgeable and undeniably the number one model at producing natural text in a large number of (human!) languages. The difference becomes especially large for more niche languages. That does not suggest a smaller model, more the opposite. The top 4 models at multilinguality are all Google : 1. 3 Pro 2. 3 Flash 3. 2.5 Pro 4. 2.5 Flash. Even the biggest OpenAI and Anthropic models can't compete in that dimension. It's definitely weaker at math and much worse at agentic things. Gemini chat as an app is also lightyears behind, it's barely different from ChatGPT at release over 3 yeaes ago. These things make it feel much weaker than it is.

algoth1 6mo ago
Aistudio should be their default app
orbital-decay 6mo ago
Regarding Anthropic, they used to make best multilingual and generalist models, it's their policy thing, not a capability issue. Claude 3 was best at this, including dead and low-resource languages. Neither modern Claude nor Gemini are remotely close to what Claude 3 was capable of (e.g. zero-shot writing styles). Anthropic basically reversed their "character training" policy and started optimizing their models for code generation at the cost of everything else, starting with Sonnet 3.5. Claude 4 took a huge hit in multilingual ability GPT, on the other hand, was always terrible at languages, except for the short-lived gpt-4.5-preview. All modern models including Gemini have bugs in basic language coherency - random language switching, self-correction attempts resulting in hallucinations etc. I speculate it's a problem with heavy RL with rewards and policies not optimized for creative writing.
- awongh 6mo ago
  The benchmarks don’t seem to say that language ability has gotten worse?
  - orbital-decay 6mo ago
    That's the thing with benchmarks, without evals and actual hands-on experience they can give you false confidence. Claude now sounds almost clinical, and is unable to speak in different styles as easily. Claude 4+ uses a lot more constructions borrowed from English than Claude 3, especially in Slavic languages where they sound unnatural. And most modern models eventually glitch out in longer texts, spitting a few garbage tokens in a random language (Telugu, Georgian, Ukrainian, totally unrelated), then continuing in the main language like nothing happened. It's rare but it happens. Samplers do not help with this, you need a second run to spellcheck it. This wasn't a problem in older models, it's a widespread issue that roughly correlates with the introduction of reasoning. Another new failure mode is self-correction in complicated texts that need reading comprehension: if the model hallucinates an incorrect fact and spots it, it tries to justify or explain it immediately. Which is much more awkward than leaving it incorrect, and also those hallucinations are more common now (maybe because the model learns to make those mistakes together with the correction? I don't know.)
    - awongh 6mo ago
      Not disputing this might be true, but this seems like something that should be capturable in a multi-lingual benchmark. Maybe it's just something that people aren't bothered with?
      - orbital-decay 6mo ago
        Basically everyone who experiments with creative writing is keenly aware of that (e.g. roleplayers), it's just the devs that have the experience training the models for it (Anthropic, DeepMind) aren't bothered doing this anymore since there's no money in it. >this seems like something that should be capturable in a multi-lingual benchmark Creative writing benchmarks just don't have good objectives to measure against. In particular, valid but inauthentic language constructions can't be captured well if your LLM judge lacks fidelity to capture it to begin with. Which is I think what typically happens. An easy litmus test would be making a selected character in a story speak Ebonics or Haitian Creole or TikTok. Claude 3 Opus was light years ahead of any model in authenticity in using them, and it was immediately obvious in a side-by-side comparison with any model including Claude 3.5+. Nuances of Polish or Russian profanities/mat or British obscenities are always the hardest for any model (they tend to either swear like dockers or tone it down, lacking the eloquence), but Opus 3 was also ahead in any of those.
ahmadyan 6mo ago
generally speaking ultra ~ mythos ~ gpt-4.5 ~ 4x behemoth pro ~ opus ~ 2x maverick flash ~ sonnet ~ scout ~ other 20-30b active Chinese models
blueblisters 6mo ago
3/3.1 Pro appears to have knowledge about eccentric topics with no obvious sources that often turns out to be right. It does hallucinate a lot though, and is the most affected by context rot in multi-turn conversations
- deaux 6mo ago
  Agreed on both, especially hallucination. That's what makes their chat app even worse, it's very opaque about web search and sources, so you can't tell whether it's a hallucination.

by deaux 6mo ago

algoth1 6mo ago

Aistudio should be their default app

orbital-decay 6mo ago

Regarding Anthropic, they used to make best multilingual and generalist models, it's their policy thing, not a capability issue. Claude 3 was best at this, including dead and low-resource languages. Neither modern Claude nor Gemini are remotely close to what Claude 3 was capable of (e.g. zero-shot writing styles). Anthropic basically reversed their "character training" policy and started optimizing their models for code generation at the cost of everything else, starting with Sonnet 3.5. Claude 4 took a huge hit in multilingual ability GPT, on the other hand, was always terrible at languages, except for the short-lived gpt-4.5-preview. All modern models including Gemini have bugs in basic language coherency - random language switching, self-correction attempts resulting in hallucinations etc. I speculate it's a problem with heavy RL with rewards and policies not optimized for creative writing.

awongh 6mo ago
The benchmarks don’t seem to say that language ability has gotten worse?
- orbital-decay 6mo ago
  That's the thing with benchmarks, without evals and actual hands-on experience they can give you false confidence. Claude now sounds almost clinical, and is unable to speak in different styles as easily. Claude 4+ uses a lot more constructions borrowed from English than Claude 3, especially in Slavic languages where they sound unnatural. And most modern models eventually glitch out in longer texts, spitting a few garbage tokens in a random language (Telugu, Georgian, Ukrainian, totally unrelated), then continuing in the main language like nothing happened. It's rare but it happens. Samplers do not help with this, you need a second run to spellcheck it. This wasn't a problem in older models, it's a widespread issue that roughly correlates with the introduction of reasoning. Another new failure mode is self-correction in complicated texts that need reading comprehension: if the model hallucinates an incorrect fact and spots it, it tries to justify or explain it immediately. Which is much more awkward than leaving it incorrect, and also those hallucinations are more common now (maybe because the model learns to make those mistakes together with the correction? I don't know.)
  - awongh 6mo ago
    Not disputing this might be true, but this seems like something that should be capturable in a multi-lingual benchmark. Maybe it's just something that people aren't bothered with?
    - orbital-decay 6mo ago
      Basically everyone who experiments with creative writing is keenly aware of that (e.g. roleplayers), it's just the devs that have the experience training the models for it (Anthropic, DeepMind) aren't bothered doing this anymore since there's no money in it. >this seems like something that should be capturable in a multi-lingual benchmark Creative writing benchmarks just don't have good objectives to measure against. In particular, valid but inauthentic language constructions can't be captured well if your LLM judge lacks fidelity to capture it to begin with. Which is I think what typically happens. An easy litmus test would be making a selected character in a story speak Ebonics or Haitian Creole or TikTok. Claude 3 Opus was light years ahead of any model in authenticity in using them, and it was immediately obvious in a side-by-side comparison with any model including Claude 3.5+. Nuances of Polish or Russian profanities/mat or British obscenities are always the hardest for any model (they tend to either swear like dockers or tone it down, lacking the eloquence), but Opus 3 was also ahead in any of those.

ahmadyan 6mo ago

generally speaking ultra ~ mythos ~ gpt-4.5 ~ 4x behemoth pro ~ opus ~ 2x maverick flash ~ sonnet ~ scout ~ other 20-30b active Chinese models

blueblisters 6mo ago

3/3.1 Pro appears to have knowledge about eccentric topics with no obvious sources that often turns out to be right. It does hallucinate a lot though, and is the most affected by context rot in multi-turn conversations

deaux 6mo ago
Agreed on both, especially hallucination. That's what makes their chat app even worse, it's very opaque about web search and sources, so you can't tell whether it's a hallucination.