5 ms·
This is for inference right? Not training?
by richwater 2y ago
This is for inference right? Not training?
- pizza 2y agoIt's for KV caching. In most conversations that will mean inference. But you can do reinforcement learning using sampled sequences, and you could use KV caching to speed that up too, so that would be an instance where training could get a slight boost.
- lawlessone 2y agodoesn't training require inference? so i guess it would help there too?
- fzzzy 2y agoTraining doesn't require inference. It uses back-propagation, a different algorithm.
- bitvoid 2y agoBackpropagation happens after some number of inferences. You need to infer to calculate a loss function to then backprop from.
- boringg 2y agoYeah but training requires the larger memory deployment data center infra