5 ms·

The only OpenAI 'crap' being used here is to generate the embeddings. Right now, OpenAI has some of the best and cheapest embeddings possible, especially for pe

by trolan 3y ago

The only OpenAI 'crap' being used here is to generate the embeddings. Right now, OpenAI has some of the best and cheapest embeddings possible, especially for personal projects. Once the vectors are created tho, you're completely off the cloud if you so choose. You can always swap out the embedding generator too, because LangChain abstracts that for your exact gripes. Everything else is already using huggingface here and can be swapped out for any other model besides GPT2 which supports the prompts.

space_fountain 3y ago
Do you have citations on OpenAI embeddings being some of the cheapest and best? The signs I've seen points almost in the opposite direction?
- colobas 3y ago
  Can you elucidate on what those signs are? Thanks in advance
  - muggermuch 3y ago
    As per the Massive Text Embedding Benchmark (MTEB) Leaderboard maintained by Huggingface, OpenAI's embedding models are not the best. https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard Of course, that's far from saying that they're the worst, or even headed that way. Just not the best (those would be a couple of fully opensource models, including those of the Instructor family, which we use at my workplace).
- ben_w 3y ago
  The only embeddings I currently see listed on https://openai.com/pricing https://openai.com/pricing are Ada v2, at $0.1/million tokens. Even if the alternative is free, how much do you value your time, how long will it take to set up an alternative, and how much use will you get out of it? If you're getting less than a million tokens and it takes half an hour longer to set up, you'd better be a student with zero literally income because that cost of time matches the UN abject poverty level. This is also why it's never been the year of linux on the desktop, and why most businesses still don't use Libre Office and GIMP. I can't speak for quality; even if I used that API directly, this whole area is changing too fast to really keep up.
  - akiselev 3y ago
    If you look at a embeddings leaderboard [1], one of the top competitors called InstructorXL [2] is just a pip install away. It's neck and neck with Ada v2 except for a shorter input length and half the dimensions, with the added benefit that you'll always have the model available. Most of the other options just work with the transformers library. [1] https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard [2] https://github.com/HKUNLP/instructor-embedding https://github.com/HKUNLP/instructor-embedding
  - arrowsmith 3y ago
    Running models on your own hardware isn't just about cost, there are privacy concerns too.
  - rolisz 3y ago
    If you've never coded or used Python before, yeah, go with OpenAI. Otherwise, generating embeddings with SentenceBERT takes 5 minutes. And from my personal experience Ada embeddings are not the best. They are large (makes aproximate searching harder), are distributed weirdly, and zimply put, other embeddings give better results for retrieval. Another advantage is that you are not an OA's whim: they just announced the deprecation of some previous model. What are you going to do when they will deprecate Ada v2 and you've built a huge system on top of it? You'll have to regenerate embeddings and hope everything still works just as well.
    - space_fountain 3y ago
      Yes, exactly this, I also want to say I'm not someone who generally thinks open models are better. I think embeddings just haven't been a focus for OpenAI and it shows. Maybe in the future they will focus on it
    - trolan 3y ago
      Yes, when new technology comes you may need to upgrade. OpenAI aren't hero's but they are covering the cost to move people from old to new embedding models
  - 3y ago
- pmichaud 3y ago
  I'm interested if you have specific options you think are better and/or cheaper.
- trolan 3y ago
  No, I have anecdotal evidence using it. How does it compare in your usage of OpenAI and competitors?
throwaway675309 3y ago
What? It's only one file, and it definitely looks like it's using openAI to make the actual queries. qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0.1), db.as_retriever())
- sp332 3y ago
  You can change the arguments to from_llm() to point to a local model instead. Example here: https://huggingface.co/TheBloke/MPT-7B-Instruct-GGML/discussions/2 https://huggingface.co/TheBloke/MPT-7B-Instruct-GGML/discuss...
aledalgrande 3y ago
> Once the vectors are created tho, you're completely off the cloud if you so choose. Ehr no? You'll need to also create an embedding of your query, which makes you totally dependent on OpenAI. If you swap out embedding algorithm you will have to regenerate all the embeddings as well, they might not be even the same size.
- trolan 3y ago
  Ah, I see what the top comment was implying. I was a bit short sighted on that side. Yes, you'd be tied to OpenAI for any new queries you need to generate. There could be some ways to offload that (vector of a vector) but it is a cloud dependency. I'd argue not a cost dependency based on how cheap these are.

by trolan 3y ago

space_fountain 3y ago

Do you have citations on OpenAI embeddings being some of the cheapest and best? The signs I've seen points almost in the opposite direction?

colobas 3y ago
Can you elucidate on what those signs are? Thanks in advance
- muggermuch 3y ago
  As per the Massive Text Embedding Benchmark (MTEB) Leaderboard maintained by Huggingface, OpenAI's embedding models are not the best. https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard Of course, that's far from saying that they're the worst, or even headed that way. Just not the best (those would be a couple of fully opensource models, including those of the Instructor family, which we use at my workplace).
ben_w 3y ago
The only embeddings I currently see listed on https://openai.com/pricing https://openai.com/pricing are Ada v2, at $0.1/million tokens. Even if the alternative is free, how much do you value your time, how long will it take to set up an alternative, and how much use will you get out of it? If you're getting less than a million tokens and it takes half an hour longer to set up, you'd better be a student with zero literally income because that cost of time matches the UN abject poverty level. This is also why it's never been the year of linux on the desktop, and why most businesses still don't use Libre Office and GIMP. I can't speak for quality; even if I used that API directly, this whole area is changing too fast to really keep up.
- akiselev 3y ago
  If you look at a embeddings leaderboard [1], one of the top competitors called InstructorXL [2] is just a pip install away. It's neck and neck with Ada v2 except for a shorter input length and half the dimensions, with the added benefit that you'll always have the model available. Most of the other options just work with the transformers library. [1] https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard [2] https://github.com/HKUNLP/instructor-embedding https://github.com/HKUNLP/instructor-embedding
- arrowsmith 3y ago
  Running models on your own hardware isn't just about cost, there are privacy concerns too.
- rolisz 3y ago
  If you've never coded or used Python before, yeah, go with OpenAI. Otherwise, generating embeddings with SentenceBERT takes 5 minutes. And from my personal experience Ada embeddings are not the best. They are large (makes aproximate searching harder), are distributed weirdly, and zimply put, other embeddings give better results for retrieval. Another advantage is that you are not an OA's whim: they just announced the deprecation of some previous model. What are you going to do when they will deprecate Ada v2 and you've built a huge system on top of it? You'll have to regenerate embeddings and hope everything still works just as well.
  - space_fountain 3y ago
    Yes, exactly this, I also want to say I'm not someone who generally thinks open models are better. I think embeddings just haven't been a focus for OpenAI and it shows. Maybe in the future they will focus on it
  - trolan 3y ago
    Yes, when new technology comes you may need to upgrade. OpenAI aren't hero's but they are covering the cost to move people from old to new embedding models
- 3y ago
pmichaud 3y ago
I'm interested if you have specific options you think are better and/or cheaper.
trolan 3y ago
No, I have anecdotal evidence using it. How does it compare in your usage of OpenAI and competitors?

throwaway675309 3y ago

What? It's only one file, and it definitely looks like it's using openAI to make the actual queries. qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0.1), db.as_retriever())

sp332 3y ago
You can change the arguments to from_llm() to point to a local model instead. Example here: https://huggingface.co/TheBloke/MPT-7B-Instruct-GGML/discussions/2 https://huggingface.co/TheBloke/MPT-7B-Instruct-GGML/discuss...

aledalgrande 3y ago

> Once the vectors are created tho, you're completely off the cloud if you so choose. Ehr no? You'll need to also create an embedding of your query, which makes you totally dependent on OpenAI. If you swap out embedding algorithm you will have to regenerate all the embeddings as well, they might not be even the same size.

trolan 3y ago
Ah, I see what the top comment was implying. I was a bit short sighted on that side. Yes, you'd be tied to OpenAI for any new queries you need to generate. There could be some ways to offload that (vector of a vector) but it is a cloud dependency. I'd argue not a cost dependency based on how cheap these are.