2 ms·

This makes no sense. It's not like they have a "slow it down" knob, they're probably parallelizing your request so you get a 2.5x speedup at 10x the price.

by stavros 9mo ago

brookst 9mo ago
All of these systems use massive pools of GPUs, and allocate many requests to each node. The “slow it down” knob is to steer a request to nodes with more concurrent requests; “speed it up” is to route to less-loaded nodes.
- stavros 9mo ago
  Right, but that's still not Anthropic adding an intentional delay for the sole purpose of having you pay more to remove it.
  - brookst 9mo ago
    Oh, of course. That’s just conspiratorial thinking. Paying to be in a premium pool makes sense, all of this “they probably serve rotten food to make people pay for quality food” nonsense is just silly.
  - yunohn 9mo ago
    But it’s actually not so difficult is it? The simplest way to make a slow pool is by having fewer GPUs and queuing requests for the non-premium users. Dead simple engineering.
    - stavros 9mo ago
      No, the simplest way is `sleep(10)`.
      - yunohn 9mo ago
        No, that wastes actual GPU resources and money. The method I described saves them money on the cheaper pool. Regardless, I sense you’re being sarcastic and difficult, so I have no desire to discuss this further.
landl0rd 9mo ago
What they are probably doing is speculative decoding, given they've mentioned identical distribution at 2.5x speed. That's roughly in the range you'd expect for that to achieve; 10x is not. It's also absolute highway robbery (or at least overly-aggressive price discrimination) to charge 6x for speculative decoding, by the way. It is not that expensive and (under certain conditions, usually very cheap drafter and high acceptance rate) actually decrease total cost. In any case, it's unlikely to be even a 2x cost increase, let alone 6x.

2 ms·

This makes no sense. It's not like they have a "slow it down" knob, they're probably parallelizing your request so you get a 2.5x speedup at 10x the price.

by stavros 9mo ago

This makes no sense. It's not like they have a "slow it down" knob, they're probably parallelizing your request so you get a 2.5x speedup at 10x the price.

brookst 9mo ago
All of these systems use massive pools of GPUs, and allocate many requests to each node. The “slow it down” knob is to steer a request to nodes with more concurrent requests; “speed it up” is to route to less-loaded nodes.
- stavros 9mo ago
  Right, but that's still not Anthropic adding an intentional delay for the sole purpose of having you pay more to remove it.
  - brookst 9mo ago
    Oh, of course. That’s just conspiratorial thinking. Paying to be in a premium pool makes sense, all of this “they probably serve rotten food to make people pay for quality food” nonsense is just silly.
  - yunohn 9mo ago
    But it’s actually not so difficult is it? The simplest way to make a slow pool is by having fewer GPUs and queuing requests for the non-premium users. Dead simple engineering.
    - stavros 9mo ago
      No, the simplest way is `sleep(10)`.
      - yunohn 9mo ago
        No, that wastes actual GPU resources and money. The method I described saves them money on the cheaper pool. Regardless, I sense you’re being sarcastic and difficult, so I have no desire to discuss this further.
landl0rd 9mo ago
What they are probably doing is speculative decoding, given they've mentioned identical distribution at 2.5x speed. That's roughly in the range you'd expect for that to achieve; 10x is not. It's also absolute highway robbery (or at least overly-aggressive price discrimination) to charge 6x for speculative decoding, by the way. It is not that expensive and (under certain conditions, usually very cheap drafter and high acceptance rate) actually decrease total cost. In any case, it's unlikely to be even a 2x cost increase, let alone 6x.