6 ms·
I think embeddings are actually low-'resolution' representations. Like GPT's ability to parse and calculate the structure of the sentence "Hello, how are you?"
by firasd 4y ago
I think embeddings are actually low-'resolution' representations. Like GPT's ability to parse and calculate the structure of the sentence "Hello, how are you?" is not represented in the embedding for the sentence. The embedding is a 1-dimensional vector and inside the model it interacts with other texts in 10,000+ dimensions
- neuronexmachina 4y agoI wonder if there'd be any use for a "ontological" representation, somewhere in-between a natural-language string and its embedding in a particular LLM. Maybe something that balances human-readability, LLM-composability, lack of brittleness, insight into the local structure of the embedding, etc.
- cs702 4y agoI wonder too. I imagine the best we could do with present technology is to get back the generated text in the form of text tokens accompanied by their corresponding deep embeddings (last hidden states): `[(text_token, deep_emb), (text_token, deep_emb), ...]`. Those deep embeddings incorporate "everything the model knows" about each generated token of text.
- neuronexmachina 4y agoMaybe a mapping/representation for a "medium embedding" could be learned that strikes a balance between shallow and deep. I have no idea what a good objective-function would be, though.
- cs702 4y agoI mean deep embeddings (i.e., sequences of hidden states, the ones are computed by all those interactions) , not the shallow embeddings of token ids in the first layer of the model! Those deep embeddings are much richer representations. Imagine if you and others building apps had access to "GPT3 deep sequence embeddings v1.0" via an API.
- machiaweliczny 4y agoIt’t that precisely embeddings API from OpenAI? It has all context so very useful for search
- cs702 4y agoNot quite. My understanding is that OpenAI's various embeddings APIs return only a single vector per document, instead of the sequence of hidden states corresponding to each predicted next token in the response generated by a GPT-type LLM. Imagine getting generated text from a GPT LLM that comes with a deep embedding of each generated token's "contextual meaning": [(text_token, deep_emb), (text_token, deep_emb), ...] allowing higher-level models and apps to use all the information in those rich representations as inputs.