4 ms·

I haven't been paying attention, why are embeddings not needed anymore?

lazzlazzlazz 3y ago
OP is incorrect. Embeddings are still needed since (1) context windows can't contain all data and (2) data memorization and continuous retraining is not yet viable.
- nextworddev 3y ago
  "yet"
  - coding123 3y ago
    It's also much slower. LLMs are generating text token at a time. That's not very good for search. Pre-search tokenization however, probably a good fit for LLMs.
- zwily 3y ago
  But the common use case of using a vector DB to pull in augmentation appears to now be handled by the Assistants API. I haven't dug into the details yet but it appears you can upload files and the contents will be used (likely with some sort of vector searching happening behind the scenes).
emadabdulrahim 3y ago
I believe their API can be stateful now: https://openai.com/blog/new-models-and-developer-products-announced-at-devday#:~:text=A%20key%20change%20introduced%20by%20this%20API%20is%20persistent%20and%20infinitely%20long%20threads%2C%20which%20allow%20developers%20to%20hand%20off%20thread%20state%20management%20to%20OpenAI%20and%20work%20around%20context%20window%20constraints.%20With%20the%20Assistants%20API%2C%20you%20simply%20add%20each%20new%20message%20to%20an%20existing%20thread https://openai.com/blog/new-models-and-developer-products-an...
sharemywin 3y ago
Retrieval: augments the assistant with knowledge from outside our models, such as proprietary domain data, product information or documents provided by your users. This means you don’t need to compute and store embeddings for your documents, or implement chunking and search algorithms. The Assistants API optimizes what retrieval technique to use based on our experience building knowledge retrieval in ChatGPT. The model then decides when to retrieve content based on the user Messages. The Assistants API automatically chooses between two retrieval techniques: it either passes the file content in the prompt for short documents, or performs a vector search for longer documents Retrieval currently optimizes for quality by adding all relevant content to the context of model calls. We plan to introduce other retrieval strategies to enable developers to choose a different tradeoff between retrieval quality and model usage cost.
- sjnair96 3y ago
  Really cool to see the Assistants API's nuanced document retrieval methods. Do you index over the text besides chunking it up and generating embeddings? I'm curious about the indexing and the depth of analysis for longer docs, like assessing an author's tone chapter by chapter—vector search might have its limits there. Plus, the process to shape user queries into retrievable embeddings seems complex. Eager to hear more about these strategies, at least what you can spill!
- riku_iki 3y ago
  > or performs a vector search for longer documents so, clients upload all their docs to OpenAI database?..
karmasimida 3y ago
Embedding is poor man's context length increase. It essentially increases your context length but with loss. There is a cost argument to make still, embedding-based approach will be cheaper and faster, but worse result than full text. That being said, I don't see how those embedding startups compete with OpenAI, no one will be able to offer better embedding than OpenAI itself. It is hardly a convincing business. The elephant in the room is the open source models aren't able to match up to OpenAI models, and it is qualitative, not quantitive.
- estreeper 3y ago
  For embeddings specifically, there are multiple open source models that outperform OpenAI’s best model (text-embedding-ada-002) that you can see on the MTEB Leaderboard [1] > embedding-based approach will be cheaper and faster, but worse result than full text I’m not sure results would be worse, I think it depends on the extent to which the models are able to ignore irrelevant context, which is a problem [2]. Using retrieval can come closer to providing only relevant context. 1. https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard 2. https://arxiv.org/abs/2302.00093 https://arxiv.org/abs/2302.00093
  - karmasimida 3y ago
    > on the MTEB Leaderboard The point isn't about leaderboard. With increasing context length, the question is on whether we need embeddings or not. With longer context length, embeddings is no longer a necessity, and it lowers its value.
    - civilitty 3y ago
      For more trivial use cases, sure, but not for harder stuff like working with US law and precedent. The US Code is on the order of tens of millions of tokens and I shudder to think how many billions of tokens make up all the judicial opinions that set or interpreted precedent.

OP is incorrect. Embeddings are still needed since (1) context windows can't contain all data and (2) data memorization and continuous retraining is not yet viable.

nextworddev 3y ago
"yet"
- coding123 3y ago
  It's also much slower. LLMs are generating text token at a time. That's not very good for search. Pre-search tokenization however, probably a good fit for LLMs.
zwily 3y ago
But the common use case of using a vector DB to pull in augmentation appears to now be handled by the Assistants API. I haven't dug into the details yet but it appears you can upload files and the contents will be used (likely with some sort of vector searching happening behind the scenes).

emadabdulrahim 3y ago

I believe their API can be stateful now: https://openai.com/blog/new-models-and-developer-products-announced-at-devday#:~:text=A%20key%20change%20introduced%20by%20this%20API%20is%20persistent%20and%20infinitely%20long%20threads%2C%20which%20allow%20developers%20to%20hand%20off%20thread%20state%20management%20to%20OpenAI%20and%20work%20around%20context%20window%20constraints.%20With%20the%20Assistants%20API%2C%20you%20simply%20add%20each%20new%20message%20to%20an%20existing%20thread https://openai.com/blog/new-models-and-developer-products-an...

sharemywin 3y ago

Retrieval: augments the assistant with knowledge from outside our models, such as proprietary domain data, product information or documents provided by your users. This means you don’t need to compute and store embeddings for your documents, or implement chunking and search algorithms. The Assistants API optimizes what retrieval technique to use based on our experience building knowledge retrieval in ChatGPT. The model then decides when to retrieve content based on the user Messages. The Assistants API automatically chooses between two retrieval techniques: it either passes the file content in the prompt for short documents, or performs a vector search for longer documents Retrieval currently optimizes for quality by adding all relevant content to the context of model calls. We plan to introduce other retrieval strategies to enable developers to choose a different tradeoff between retrieval quality and model usage cost.

sjnair96 3y ago
Really cool to see the Assistants API's nuanced document retrieval methods. Do you index over the text besides chunking it up and generating embeddings? I'm curious about the indexing and the depth of analysis for longer docs, like assessing an author's tone chapter by chapter—vector search might have its limits there. Plus, the process to shape user queries into retrievable embeddings seems complex. Eager to hear more about these strategies, at least what you can spill!
riku_iki 3y ago
> or performs a vector search for longer documents so, clients upload all their docs to OpenAI database?..

karmasimida 3y ago

Embedding is poor man's context length increase. It essentially increases your context length but with loss. There is a cost argument to make still, embedding-based approach will be cheaper and faster, but worse result than full text. That being said, I don't see how those embedding startups compete with OpenAI, no one will be able to offer better embedding than OpenAI itself. It is hardly a convincing business. The elephant in the room is the open source models aren't able to match up to OpenAI models, and it is qualitative, not quantitive.

estreeper 3y ago
For embeddings specifically, there are multiple open source models that outperform OpenAI’s best model (text-embedding-ada-002) that you can see on the MTEB Leaderboard [1] > embedding-based approach will be cheaper and faster, but worse result than full text I’m not sure results would be worse, I think it depends on the extent to which the models are able to ignore irrelevant context, which is a problem [2]. Using retrieval can come closer to providing only relevant context. 1. https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard 2. https://arxiv.org/abs/2302.00093 https://arxiv.org/abs/2302.00093
- karmasimida 3y ago
  > on the MTEB Leaderboard The point isn't about leaderboard. With increasing context length, the question is on whether we need embeddings or not. With longer context length, embeddings is no longer a necessity, and it lowers its value.
  - civilitty 3y ago
    For more trivial use cases, sure, but not for harder stuff like working with US law and precedent. The US Code is on the order of tens of millions of tokens and I shudder to think how many billions of tokens make up all the judicial opinions that set or interpreted precedent.