4 ms·

In every model I’ve worked with, all of those steps are EXPLICIT. You take input, then tokenize, then embed, feed to model, then convert from embeddings to toke

In every model I’ve worked with, all of those steps are EXPLICIT. You take input, then tokenize, then embed, feed to model, then convert from embeddings to tokens, then back to words. There’s actually a lot of work done in embedding space! Chained execution, neural graph reasoning (see https://arxiv.org/pdf/2303.14617.pdf https://arxiv.org/pdf/2303.14617.pdf), even basic IR tasks like clustering and retrieval. There’s been no real effort to especially expose it because that is what you get by default. Even OpenAI has an “embed” endpoint. So you’re not going to see a huge push for it the same way you won’t see a push for reasoning about websites “in the HTML”. :)

sp332 3y ago
I only see a way to get embeddings out. How do I put embeddings into an OpenAI API?
- dustymcp 3y ago
  im not sure i follow if you want embeddings for any word/sentence you send it to the embeddingendpoint and save the vectors coming back?
  - TeMPOraL 3y ago
    The point is that we want to shove arbitrary embeddings vector into an LLM and inspect what comes out. This is different from feeding tokens as input. The kind of thing I'd like to do, for example, is to: - Get embeddings for e.g. "blue" and "red", or "the sky is blue" and "galaxy redshift"; - Average them, resulting a vector that's bound to not be expressible with tokens alone; - Input that to the same model I got the embeddings from, and see what comes out. If by "embeddingendpoint" you mean OpenAI, they provide that for a specialized model, and (AFAIK) you can only get embeddings out (for the purpose of comparing various vectors yourself). They have no API endpoint for an LLM that can take those embeddings as input.
    - freeone3000 3y ago
      Run the resultant embeddings through local beamsearch to get tokens, then run the detokenizer to get back words.
- macrolime 3y ago
  You can't and you can only get embeddings from a tiny gpt-3 model.

sp332 3y ago

I only see a way to get embeddings out. How do I put embeddings into an OpenAI API?

dustymcp 3y ago
im not sure i follow if you want embeddings for any word/sentence you send it to the embeddingendpoint and save the vectors coming back?
- TeMPOraL 3y ago
  The point is that we want to shove arbitrary embeddings vector into an LLM and inspect what comes out. This is different from feeding tokens as input. The kind of thing I'd like to do, for example, is to: - Get embeddings for e.g. "blue" and "red", or "the sky is blue" and "galaxy redshift"; - Average them, resulting a vector that's bound to not be expressible with tokens alone; - Input that to the same model I got the embeddings from, and see what comes out. If by "embeddingendpoint" you mean OpenAI, they provide that for a specialized model, and (AFAIK) you can only get embeddings out (for the purpose of comparing various vectors yourself). They have no API endpoint for an LLM that can take those embeddings as input.
  - freeone3000 3y ago
    Run the resultant embeddings through local beamsearch to get tokens, then run the detokenizer to get back words.
macrolime 3y ago
You can't and you can only get embeddings from a tiny gpt-3 model.