3 ms·

Latency isn't performance? Maybe you mean "not throughput-critical"?

• rockwotj 2mo ago
  but according to Little’s law, if you improve latency, you also improve throughput, right? If I have the same number of CPU cores and they all can do their work in half the time they can double the number of requests now
  • scheme271 2mo ago
    I don't think that's accurate. If tokenization takes say 10ms and the rest of the inference steps take 50 ms then, improving tokenization will improve the time to first token but won't affect throughput much. After the first token, the inference steps effectively hide the tokenization time.
  • ludston 2mo ago
    It depends on whether or not latency is the constraint of throughput in your circumstances.
• jaggederest 2mo ago
  Just to clarify, latency is one form of performance, and a separate thing to optimize from total resource usage in more classic "performance critical" situations. That performance might be energy, space, or other dimensions besides latency. It might also be something like reliability, accuracy, precision, or even the very human factors like simplicity, modifiability, and visibility. Heck, even latency alone you can just reduce the standard deviation and get smoother flows. Little's law is a great callout here too, one of my favorite computer science principles.