4 ms·

I was thinking about inhouse model inference speeds at frontier labs like Anthropic and OpenAI after reading the "Claude built a C compiler" article. Having hi

by jawon 8mo ago

I was thinking about inhouse model inference speeds at frontier labs like Anthropic and OpenAI after reading the "Claude built a C compiler" article. Having higher inference speed would be an advantage, especially if you're trying to eat all the software and services. Anthropic offering 2.5x makes me assume they have 5x or 10x themselves. In the predicted nightmare future where everything happens via agents negotiating with agents, the side with the most compute, and the fastest compute, is going to steamroll everyone.

crowbahr 8mo ago
Where on earth are you getting these numbers? Why would a SaaS company that is fighting for market dominance withhold 10x performance if they had it? Where are you getting 2.5x? This is such bizarre magical thinking, borderline conspiratorial. There is no reason to believe any of the big AI players are serving anything less than the best trade off of stability and speed that they can possibly muster, especially when their cost ratios are so bad.
- jawon 8mo ago
  Not magical thinking, not conspiratorial, just hypothetical. Just because you can't afford to 10x all your customers' inference doesn't mean you can't afford to 10x your inhouse inference. And 2.5x is from Anthropic's latest offering. But it costs you 6x normal API pricing.
  - jawon 8mo ago
    Also, from a comment in another thread, from roon, who works at OpenAI: > codex-5.2 is really amazing but using it from my personal and not work account over the weekend taught me some user empathy lol it’s a bit slow [0] https://nitter.net/tszzl/status/2016338961040548123 https://nitter.net/tszzl/status/2016338961040548123
- deleted 8mo ago
  [deleted]
falloutx 8mo ago
Thats also called slowing down default experience so users have to pay more for the fast mode. I think its the first time we are seeing blatant speed ransoms in the LLMs.
- throw310822 8mo ago
  Slowing down respect to what?
  - falloutx 8mo ago
    Slowing down with respect to original speed of response. Basically what we used to get few months back and what is the best possible experience.
    - throw310822 8mo ago
      There is no "original speed of response". The more resources you pour in, the faster it goes.
      - falloutx 8mo ago
        Watch them decrease resources for the normal mode so people are penny pinched into using fast mode.
        throw310822 8mo ago
        Seriously, thinking at the price structure of this (6x the price for 2.5x the speed, if that's correct) it seems to target something like real time applications with very small context. Maybe vocal assistants? I guess that if you're doing development it makes more sense to parallelize over more agents rather than paying that much for a modest increase in speed.
- Aurornis 8mo ago
  That's not how this works. LLM serving at scale processes multiple requests in parallel for efficiency. Reduce the parallelism and you can process individual requests faster, but the overall number of tokens processed is lower.
stavros 8mo ago
This makes no sense. It's not like they have a "slow it down" knob, they're probably parallelizing your request so you get a 2.5x speedup at 10x the price.
- brookst 8mo ago
  All of these systems use massive pools of GPUs, and allocate many requests to each node. The “slow it down” knob is to steer a request to nodes with more concurrent requests; “speed it up” is to route to less-loaded nodes.
  - stavros 8mo ago
    Right, but that's still not Anthropic adding an intentional delay for the sole purpose of having you pay more to remove it.
    - brookst 8mo ago
      Oh, of course. That’s just conspiratorial thinking. Paying to be in a premium pool makes sense, all of this “they probably serve rotten food to make people pay for quality food” nonsense is just silly.
    - yunohn 8mo ago
      But it’s actually not so difficult is it? The simplest way to make a slow pool is by having fewer GPUs and queuing requests for the non-premium users. Dead simple engineering.
      - stavros 8mo ago
        No, the simplest way is `sleep(10)`.
        yunohn 8mo ago
        No, that wastes actual GPU resources and money. The method I described saves them money on the cheaper pool. Regardless, I sense you’re being sarcastic and difficult, so I have no desire to discuss this further.
Aurornis 8mo ago
> Anthropic offering 2.5x makes me assume they have 5x or 10x themselves. They said the 2.5X offering is what they've been using internally. Now they're offering via the API: https://x.com/claudeai/status/2020207322124132504 https://x.com/claudeai/status/2020207322124132504 LLM APIs are tuned to handle a lot of parallel requests. In short, the overall token throughput is higher, but the individual requests are processed more slowly. The scaling curves aren't that extreme, though. I doubt they could tune the knobs to get individual requests coming through at 10X the normal rate. This likely comes from having some servers tuned for higher individual request throughput, at the expense of overall token throughput. It's possible that it's on some newer generation serving hardware, too.

by jawon 8mo ago

crowbahr 8mo ago

Where on earth are you getting these numbers? Why would a SaaS company that is fighting for market dominance withhold 10x performance if they had it? Where are you getting 2.5x? This is such bizarre magical thinking, borderline conspiratorial. There is no reason to believe any of the big AI players are serving anything less than the best trade off of stability and speed that they can possibly muster, especially when their cost ratios are so bad.

jawon 8mo ago
Not magical thinking, not conspiratorial, just hypothetical. Just because you can't afford to 10x all your customers' inference doesn't mean you can't afford to 10x your inhouse inference. And 2.5x is from Anthropic's latest offering. But it costs you 6x normal API pricing.
- jawon 8mo ago
  Also, from a comment in another thread, from roon, who works at OpenAI: > codex-5.2 is really amazing but using it from my personal and not work account over the weekend taught me some user empathy lol it’s a bit slow [0] https://nitter.net/tszzl/status/2016338961040548123 https://nitter.net/tszzl/status/2016338961040548123
deleted 8mo ago
[deleted]

falloutx 8mo ago

Thats also called slowing down default experience so users have to pay more for the fast mode. I think its the first time we are seeing blatant speed ransoms in the LLMs.

throw310822 8mo ago
Slowing down respect to what?
- falloutx 8mo ago
  Slowing down with respect to original speed of response. Basically what we used to get few months back and what is the best possible experience.
  - throw310822 8mo ago
    There is no "original speed of response". The more resources you pour in, the faster it goes.
    - falloutx 8mo ago
      Watch them decrease resources for the normal mode so people are penny pinched into using fast mode.
      - throw310822 8mo ago
        Seriously, thinking at the price structure of this (6x the price for 2.5x the speed, if that's correct) it seems to target something like real time applications with very small context. Maybe vocal assistants? I guess that if you're doing development it makes more sense to parallelize over more agents rather than paying that much for a modest increase in speed.
Aurornis 8mo ago
That's not how this works. LLM serving at scale processes multiple requests in parallel for efficiency. Reduce the parallelism and you can process individual requests faster, but the overall number of tokens processed is lower.

stavros 8mo ago

This makes no sense. It's not like they have a "slow it down" knob, they're probably parallelizing your request so you get a 2.5x speedup at 10x the price.

brookst 8mo ago
All of these systems use massive pools of GPUs, and allocate many requests to each node. The “slow it down” knob is to steer a request to nodes with more concurrent requests; “speed it up” is to route to less-loaded nodes.
- stavros 8mo ago
  Right, but that's still not Anthropic adding an intentional delay for the sole purpose of having you pay more to remove it.
  - brookst 8mo ago
    Oh, of course. That’s just conspiratorial thinking. Paying to be in a premium pool makes sense, all of this “they probably serve rotten food to make people pay for quality food” nonsense is just silly.
  - yunohn 8mo ago
    But it’s actually not so difficult is it? The simplest way to make a slow pool is by having fewer GPUs and queuing requests for the non-premium users. Dead simple engineering.
    - stavros 8mo ago
      No, the simplest way is `sleep(10)`.
      - yunohn 8mo ago
        No, that wastes actual GPU resources and money. The method I described saves them money on the cheaper pool. Regardless, I sense you’re being sarcastic and difficult, so I have no desire to discuss this further.

Aurornis 8mo ago

> Anthropic offering 2.5x makes me assume they have 5x or 10x themselves. They said the 2.5X offering is what they've been using internally. Now they're offering via the API: https://x.com/claudeai/status/2020207322124132504 https://x.com/claudeai/status/2020207322124132504 LLM APIs are tuned to handle a lot of parallel requests. In short, the overall token throughput is higher, but the individual requests are processed more slowly. The scaling curves aren't that extreme, though. I doubt they could tune the knobs to get individual requests coming through at 10X the normal rate. This likely comes from having some servers tuned for higher individual request throughput, at the expense of overall token throughput. It's possible that it's on some newer generation serving hardware, too.