3 ms·

All of these systems use massive pools of GPUs, and allocate many requests to each node. The “slow it down” knob is to steer a request to nodes with more concur

by brookst 9mo ago

All of these systems use massive pools of GPUs, and allocate many requests to each node. The “slow it down” knob is to steer a request to nodes with more concurrent requests; “speed it up” is to route to less-loaded nodes.

stavros 9mo ago
Right, but that's still not Anthropic adding an intentional delay for the sole purpose of having you pay more to remove it.
- brookst 9mo ago
  Oh, of course. That’s just conspiratorial thinking. Paying to be in a premium pool makes sense, all of this “they probably serve rotten food to make people pay for quality food” nonsense is just silly.
- yunohn 9mo ago
  But it’s actually not so difficult is it? The simplest way to make a slow pool is by having fewer GPUs and queuing requests for the non-premium users. Dead simple engineering.
  - stavros 9mo ago
    No, the simplest way is `sleep(10)`.
    - yunohn 9mo ago
      No, that wastes actual GPU resources and money. The method I described saves them money on the cheaper pool. Regardless, I sense you’re being sarcastic and difficult, so I have no desire to discuss this further.