3 ms·
I don't think the normalization makes it infeasible. They should be able to make an adjustment (the reverse of the normalization) in one operation. I think they
by danielmarkbruce 2y ago
I don't think the normalization makes it infeasible. They should be able to make an adjustment (the reverse of the normalization) in one operation. I think they are caching the attention calcs.
The hard thing (I think) is what to keep in the cache and where to keep it given you are serving lots of customers and the attention calc can be a large set of numbers pretty quickly.
- jncfhnb 2y agoIt’s only “hard” because they don’t want to let customers supply the cache of course
- danielmarkbruce 2y agoI'm not sure that's it. Presumably they want to keep the cache in GPU memory?
- jncfhnb 2y agoThat’s largely it imo. If you get get the embedding representations you could just recreate the model logic and then it’s no longer closed source.
- danielmarkbruce 2y agono, you couldn't. If they just handed you the embedding layer it wouldn't help that much. Plus, I'm reasonably certain they are caching the attention scores anyway.