3 ms·

While not actually speeding things up, I wonder if it's possible to run a very small autocomplete model to guess what the next sentence ahead of time from the L

While not actually speeding things up, I wonder if it's possible to run a very small autocomplete model to guess what the next sentence ahead of time from the LLM on the client side. To at least keep the perceived latency low. Thus if there is a delay, you still get the stream of text coming at a steady rate, but you may have to backtrack and correct for mispredictions.

gwern 3y ago
Yes, that sort of speculative execution has long existed for NNs, and has enjoyed a recent resurgence in popularity as people speculate that OA has been using that heavily for its APIs & services. It can work very well for outputs that vary greatly in their predictability (like natural language), but can also backfire - nobody likes seeing the predictions thrash around unpredictably, but if you aren't careful, you accept bad predictions or wind up not saving latency or compute at all.