3 ms·

Is tokenizing really the bottleneck? If we go from 20ms to 15ms does it really matter?

• gnabgib 3mo ago
  You sound like someone who used to write fastruby
  • phplovesong 3mo ago
    Never heard of fast ruby. My point is tokenizing is rarely a bottleneck, as 99% of time is spent in inference. So you speed up 1% of the pipeline by some factor, and the end result is unobservable for a human.
    • antonvs 3mo ago
      Time to first token is observable by a human, and they’re reporting up to 10% reduction there. Plus, inference is not the only place tokenization happens. This can make a big difference during development of ML models.