4 ms·
I think a number of us are using the method and these folks decided to do a paper and release their findings.
by outofpaper 2y ago
I think a number of us are using the method and these folks decided to do a paper and release their findings.
- jimmySixDOF 2y agoLots of speedups already on Llama3 like Groq (hardware arch) and Modal [1] getting 200 t/s From the below Modal link they use: >continuous batching, so multiple generations can take place at the same time on a single container >PagedAttention, which applies memory paging to the attention mechanism’s key-value cache, increasing throughput [1]https://modal.com/docs/examples/text_generation_inference https://modal.com/docs/examples/text_generation_inference