3 ms·

Wow, I'm mindblown this isn't getting more attention. This seems like a clear win for inference. Fine tuning cost for this is reasonable (around 0.01% of the or

Wow, I'm mindblown this isn't getting more attention. This seems like a clear win for inference. Fine tuning cost for this is reasonable (around 0.01% of the original pre-training cost). And the performance wins seem fairly consistent.

lopuhin 2y ago
Similar or greater inference wins are achieved with speculative decoding which is already widely used, so while this is really interesting (and was tried before with less success AFAIK), it's not yet clear how impactful it would be.
- WhitneyLand 2y ago
  I don’t see where similar wins have ever been achieved. Speculative decoding can reduce latency, but at the cost of using a lot more compute. The amazing thing here is latency and global throughput improvements would be realized because of the increase in efficiency. From what I understand speculative decoding can also come with more challenges insofar as trying to maintain overall output quality.
snyhlxde 2y ago
Thanks for interesting in our work! Yes we found training with consistency loss + AR loss on even a subset of a dataset results in significant speedup (0.01% pre-training cost). Training on more data permits even further speedup: the model is able to learn from more frequently-appearing collocations and phrases. For more details, please check out our paper and you can also see speedup saturates as the size of training data grows.
WhitneyLand 2y ago
Yes, seems like a huge important result for LLM performance. I’m not aware of any other paper that has offered to increase inference LLM performance to this degree. Has there ever been one before? At least while also: - Maintaining output quality. The benchmarks used were somewhat narrow but so far so good. - Improving not just query latency but also global throughput - Not requiring more compute - Having a relatively practical implementation and not adding big challenges and complexity You could argue the insight is incremental, as it builds on what’s been done with parallel/jacobi decoding. Those previous results were necessary and important, but this may be the one that finally extracts real world value from the promise of parallel decoding.

lopuhin 2y ago

Similar or greater inference wins are achieved with speculative decoding which is already widely used, so while this is really interesting (and was tried before with less success AFAIK), it's not yet clear how impactful it would be.

WhitneyLand 2y ago
I don’t see where similar wins have ever been achieved. Speculative decoding can reduce latency, but at the cost of using a lot more compute. The amazing thing here is latency and global throughput improvements would be realized because of the increase in efficiency. From what I understand speculative decoding can also come with more challenges insofar as trying to maintain overall output quality.

snyhlxde 2y ago

Thanks for interesting in our work! Yes we found training with consistency loss + AR loss on even a subset of a dataset results in significant speedup (0.01% pre-training cost). Training on more data permits even further speedup: the model is able to learn from more frequently-appearing collocations and phrases. For more details, please check out our paper and you can also see speedup saturates as the size of training data grows.

WhitneyLand 2y ago

Yes, seems like a huge important result for LLM performance. I’m not aware of any other paper that has offered to increase inference LLM performance to this degree. Has there ever been one before? At least while also: - Maintaining output quality. The benchmarks used were somewhat narrow but so far so good. - Improving not just query latency but also global throughput - Not requiring more compute - Having a relatively practical implementation and not adding big challenges and complexity You could argue the insight is incremental, as it builds on what’s been done with parallel/jacobi decoding. Those previous results were necessary and important, but this may be the one that finally extracts real world value from the promise of parallel decoding.