4 ms·
The transformer only looks backwards, so if the first part of the sequence (the prompt) doesn't change, you don't need to rerun it again on that part, just on t
by logicchains 2y ago
The transformer only looks backwards, so if the first part of the sequence (the prompt) doesn't change, you don't need to rerun it again on that part, just on the part after it that changed. For use cases with large prompts relative to the output size (e.g. lots of examples in the prompt), this can significantly speed up the workload.
- jncfhnb 2y agoI think most architectures do a layer of normalization on the whole text embeddings before calculating attention which makes this infeasible Shouldn’t be a huge deal to adjust imo One of the bigger problems is that closed model providers don’t want to expose the embedding space and let’s users see what they have
- danielmarkbruce 2y agoI don't think the normalization makes it infeasible. They should be able to make an adjustment (the reverse of the normalization) in one operation. I think they are caching the attention calcs. The hard thing (I think) is what to keep in the cache and where to keep it given you are serving lots of customers and the attention calc can be a large set of numbers pretty quickly.
- jncfhnb 2y agoIt’s only “hard” because they don’t want to let customers supply the cache of course
- danielmarkbruce 2y agoI'm not sure that's it. Presumably they want to keep the cache in GPU memory?
- jncfhnb 2y agoThat’s largely it imo. If you get get the embedding representations you could just recreate the model logic and then it’s no longer closed source.
- danielmarkbruce 2y agono, you couldn't. If they just handed you the embedding layer it wouldn't help that much. Plus, I'm reasonably certain they are caching the attention scores anyway.