7 ms·
Re: step 3, I think you mean text-embedding-ada-002—OpenAI's current embeddings model, which replaces all 15 (or is it 16?) first generation embeddings models.
by npsomaratna 3y ago
Re: step 3, I think you mean text-embedding-ada-002—OpenAI's current embeddings model, which replaces all 15 (or is it 16?) first generation embeddings models.
With respect to open source embeddings models, instructor-xl is the state of the art currently—as effective as text-embedding-ada-002.
That said, instructor-xl has a context length of 512 tokens, while text-embedding-ada-002 has a context length of 8192 tokens, which is markedly more convenient.
Last but not least, parent's comment re: langchain is spot on. It's simple and straightforward to write these few lines of code yourself.
- KRAKRISMOTT 3y agoYou are right, my mistake, it should be text-embedding-ada. Thanks for the info about instructor, will check it out. What do you usually use LLMs for?
- npsomaratna 3y agoMainly for data processing, and almost exclusively "open-source" LLMs now (essential given our focus on industries with tons of confidential data). Let me plug my startup's website, which describes what we do: https://www.amw.ai/ https://www.amw.ai/
- ianpurton 3y agoFrom you home page, I couldn't figure out exactly what you do. Can you parse PDF into embeddings for a vector database?
- kirdiekirdie 3y ago[dead]