3 ms·
I mean deep embeddings (i.e., sequences of hidden states, the ones are computed by all those interactions) , not the shallow embeddings of token ids in the firs
by cs702 4y ago
I mean deep embeddings (i.e., sequences of hidden states, the ones are computed by all those interactions) , not the shallow embeddings of token ids in the first layer of the model! Those deep embeddings are much richer representations.
Imagine if you and others building apps had access to "GPT3 deep sequence embeddings v1.0" via an API.
- machiaweliczny 4y agoIt’t that precisely embeddings API from OpenAI? It has all context so very useful for search
- cs702 4y agoNot quite. My understanding is that OpenAI's various embeddings APIs return only a single vector per document, instead of the sequence of hidden states corresponding to each predicted next token in the response generated by a GPT-type LLM. Imagine getting generated text from a GPT LLM that comes with a deep embedding of each generated token's "contextual meaning": [(text_token, deep_emb), (text_token, deep_emb), ...] allowing higher-level models and apps to use all the information in those rich representations as inputs.