3 ms·
Whoa. How is that possible?
by baobabKoodaa 4y ago
Whoa. How is that possible?
- bckr 4y agoYeah, what is being stored that is available between inferences?
- mike_hearn 4y agoNothing, if by "inference" you mean single API call. The underlying algorithm starts (very roughly) by processing the prompt into a set of vectors, which are then used repeatedly to generate tokens one after the other. What's meant here is that the act of generating and using those prompt vectors has an effect sort of like regular training. It doesn't mean anything is being written to disk afterwards, though of course you could write those vectors to disk and re-use them. It just doesn't make much sense at the moment. Doing that isn't enough to give you infinite context windows.
- sharepower123 4y agoAn hypothesis is that in the latent space there is something like a branching factor that can be used in context learning to select the main tree for others layers. So a LLM is able to have the knowledge of many specialized smallers LLM and select the appropriate one by way of giving values to the branching factor. The branching factor could be some combination of attention layers operating in the latent space of previous layers.
- meh8881 4y agoRemember the ancient “king - man = queen” word vectors? This is just that but produced by a model rather than manually adding vectors and finding the nearest one.
- kgwgk 4y agoDepending on what one understands by “new passes through the model” it may not be.
- deleted 4y ago[deleted]