6 ms·
This is awesome, but tokenization is typically <0.1% of total inference time. Presumably there's a host of applications that just need to tokenize, though, and
by onlyrealcuzzo 3mo ago
This is awesome, but tokenization is typically <0.1% of total inference time.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
- GenerocUsername 3mo agoAlways good to make it 0.001%
- pipsterwo 3mo ago1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload (edit: based on the assumption above) Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-20-gartner-forecasts-worldwide-ai-platforms-and-models-market-to-grow-63-percent-in-2026 https://www.gartner.com/en/newsroom/press-releases/2026-07-2...
- boroboro4 3mo agoThe issue is it’s cpu compute which is underutilized in gpu clusters anyway, so practically it’s not really 1/1000.
- pipsterwo 3mo agoTotally, edited my comment to specify "based on the assumption above." The main takeaway I was going for was 0.1% is not a small number in this context
- scottcha 3mo agoI run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.
- jaggederest 3mo agoTo concur it's "latency critical", not "performance critical", people often confuse those two - optimize it all, but especially the chained critical path latency!
- dataflow 3mo agoLatency isn't performance? Maybe you mean "not throughput-critical"?
- rockwotj 3mo agobut according to Little’s law, if you improve latency, you also improve throughput, right? If I have the same number of CPU cores and they all can do their work in half the time they can double the number of requests now
- scheme271 3mo agoI don't think that's accurate. If tokenization takes say 10ms and the rest of the inference steps take 50 ms then, improving tokenization will improve the time to first token but won't affect throughput much. After the first token, the inference steps effectively hide the tokenization time.
- ludston 3mo agoIt depends on whether or not latency is the constraint of throughput in your circumstances.
- jaggederest 3mo agoJust to clarify, latency is one form of performance, and a separate thing to optimize from total resource usage in more classic "performance critical" situations. That performance might be energy, space, or other dimensions besides latency. It might also be something like reliability, accuracy, precision, or even the very human factors like simplicity, modifiability, and visibility. Heck, even latency alone you can just reduce the standard deviation and get smoother flows. Little's law is a great callout here too, one of my favorite computer science principles.
- brcmthrowaway 3mo ago[dead]
- noahbp 3mo agoTime to first token, especially for smaller models, can be sharply reduced. Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.
- fastball 3mo agoTokenization is <0.1% of the inference time for the first token in the same way it is <0.1% for the last.
- marcelroed 3mo agoTime to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
- dingdingdang 3mo agoHave you done preliminary numbers on replacing tokenizer on, say, llama-server?
- marcelroed 3mo agoRunning the numbers now
- marcelroed 3mo agoAdded numbers here: https://news.ycombinator.com/item?id=49015014 https://news.ycombinator.com/item?id=49015014
- marcelroed 3mo agoAuthor here: Actually, depending on the nature of the inference you're doing it can be quite significant. Here are some numbers for time-to-first-token (time to process the entire input and produce the first token of output) for an 8B Qwen3 model running on a single B200. Obviously these numbers are more significant with smaller models and on faster GPUs. Credit to fastokens [0] for the benchmark. sglang_speed [huggingface]: mean=10.31ms median=6.48ms p99=45.98ms rps=96.8 sglang_speed [gigatoken]: mean=10.13ms median=6.54ms p99=45.16ms rps=98.4 input_len= 2048: TTFT mean 30.74 -> 29.05 ms (+5.5% reduction) | median 31.00 -> 28.80 (+7.1%) | p99 33.02 -> 32.02 (+3.0%) input_len= 8192: TTFT mean 105.20 -> 96.36 ms (+8.4% reduction) | median 103.87 -> 95.49 (+8.1%) | p99 126.88 -> 113.84 (+10.3%) input_len= 32768: TTFT mean 687.05 -> 633.66 ms (+7.8% reduction) | median 708.14 -> 657.35 (+7.2%) | p99 728.95 -> 678.79 (+6.9%) These are preliminary numbers, so I will need to do some more testing before including this in the README. [0] https://github.com/crusoecloud/fastokens https://github.com/crusoecloud/fastokens
- treyd 3mo agoMy understanding is that tokenization is largely serial, so for a large initial prompt it can make up a large chunk of input processing time since after handing it off to the model inference it's (able to be) fully parallel across all tokens.
- rockinghigh 2mo agoFor small models, tokenization can reach 1-10% of total inference time.