4 ms·

Right about where it is today with better integrations? One year is the difference between Sonnet 3.5 and Opus 4.5. We're not hitting diminishing returns yet (

Right about where it is today with better integrations? One year is the difference between Sonnet 3.5 and Opus 4.5. We're not hitting diminishing returns yet (mostly because of exponential capex scaling, but still). We're already committed to ~3 years of the current trajectory, which means we can expect similar performance boosts year over year. The key to keep in mind is that LLMs are a giant bag of capabilities, and just because we hit diminishing returns on one capability, that doesn't say much if anything about your ability to scale other capabilities.

deleted 11mo ago
[deleted]
catlifeonmars 11mo ago
You buried the lede with “exponential capex scaling”. How is this technology not like oil extraction? The bulk of that capex is chips, and those chips are straight up depreciating assets.
- CuriouslyC 11mo ago
  The depreciation schedule is debatable (and that's currently a big issue!). We've been depreciating based on availability of next generation chips rather than useful life, but I've seen 8 year old research clusters with low replacement rates. If we stop spending on infra now, that would still give us an engine well into the next decade.
vrighter 11mo ago
better integrations won't do anything to fix the fact that these tools are, by their mathematical nature, unreliable and always will be
- blackqueeriroh 11mo ago
  so are people
  - vrighter 11mo ago
    But humans have vastly lower error rates than llms. And in a multi-step process that means that those error rates compound. And when that happens, you end up with a 50/50 or worse
    - danaris 11mo ago
      And, more importantly, a given human can, and usually will, learn from their mistakes and do better in a reasonably consistent pattern. And when humans do make mistakes, they're also in patterns that are fairly predictable and easy for other humans to understand, because we make mistakes due to a few different well-known categories of errors of thought and behavior. LLMs, meanwhile, make mistakes simply because they happen to have randomly generated incorrect text that time. Or, to look at it another way, they get things right simply because they happen to have randomly generated correct text that time. Individual humans can be highly reliable. Humans can consciously make tradeoffs between speed and reliability. Individual unreliable humans can become more reliable through time and effort. None of these are true of LLMs.
cpt_sobel 11mo ago
> We're already committed to ~3 years of the current trajectory How do you mean committed?

deleted 11mo ago

[deleted]

catlifeonmars 11mo ago

You buried the lede with “exponential capex scaling”. How is this technology not like oil extraction? The bulk of that capex is chips, and those chips are straight up depreciating assets.

CuriouslyC 11mo ago
The depreciation schedule is debatable (and that's currently a big issue!). We've been depreciating based on availability of next generation chips rather than useful life, but I've seen 8 year old research clusters with low replacement rates. If we stop spending on infra now, that would still give us an engine well into the next decade.

vrighter 11mo ago

better integrations won't do anything to fix the fact that these tools are, by their mathematical nature, unreliable and always will be

blackqueeriroh 11mo ago
so are people
- vrighter 11mo ago
  But humans have vastly lower error rates than llms. And in a multi-step process that means that those error rates compound. And when that happens, you end up with a 50/50 or worse
  - danaris 11mo ago
    And, more importantly, a given human can, and usually will, learn from their mistakes and do better in a reasonably consistent pattern. And when humans do make mistakes, they're also in patterns that are fairly predictable and easy for other humans to understand, because we make mistakes due to a few different well-known categories of errors of thought and behavior. LLMs, meanwhile, make mistakes simply because they happen to have randomly generated incorrect text that time. Or, to look at it another way, they get things right simply because they happen to have randomly generated correct text that time. Individual humans can be highly reliable. Humans can consciously make tradeoffs between speed and reliability. Individual unreliable humans can become more reliable through time and effort. None of these are true of LLMs.

cpt_sobel 11mo ago

> We're already committed to ~3 years of the current trajectory How do you mean committed?