3 ms·

Their OpenAI benchmark is GPT3 (text-embedding-ada-002), not GPT4.

by e1g 3y ago

simonw 3y ago
"text-embedding-ada-002" isn't GPT3, it's a different kind of model. Embedding models and Large Language Models aren't the same thing.
- e1g 3y ago
  LLMs and embedding models are certainly different, but it's a useful benchmark to calibrate expectations. OpenAI released text-embedding-ada-002 a year ago, and they describe the ada model as[1] "the original GPT-3 base model [...] capable of very simple tasks, usually the fastest model in the GPT-3 series". It's fair to expect GPT3-level results - not GPT 3.5 and certainly not open-source tiny GPT4 as some might think when they read "rivaling OpenAI". [1] https://platform.openai.com/docs/models/whisper https://platform.openai.com/docs/models/whisper
  - minimaxir 3y ago
    When people talked about GPT-3 they always referred to davinci which is the largest model, not ada.
  - simonw 3y ago
    No, you're confusing two things here. "text-ada-001" is LLM in the GPT3 family, described as "Capable of very simple tasks, usually the fastest model in the GPT-3 series, and lowest cost" "text-embedding-ada-002" is entirely different - that page describes it as "Our second generation embedding model, text-embedding-ada-002 is a designed to replace the previous 16 first-generation embedding models at a fraction of the cost."
    - minimaxir 3y ago
      tl;dr OpenAI is bad at product naming.
    - e1g 3y ago
      OpenAI doesn't say directly what text-embedding-ada-002 is, but in the release blog post they show that performance is comparable to davinci/curie, which places it firmly in the universe of GPT3. I understand it's not a straight line comparison, but to me it's still a useful mental heuristic about what to expect. [1] https://openai.com/blog/new-and-improved-embedding-model https://openai.com/blog/new-and-improved-embedding-model (see "Model improvements")
      - helloplanets 3y ago
        Reading through that article, the specific Davinci/Curie models they seem to be referring to are called the following: 'text-search-davinci-001', 'text-search-curie-001', 'text-similarity-davinci-001' and 'text-similarity-curie-001'. Are you sure these have anything to do with 'text-davinci-003' or 'text-curie-001'? Will have to agree with everyone here that OpenAI is good at being extremely confusing. It seems like the logic might be something along the lines of the 'text-search' portion being the actual type of the model, while the 'curie-001' / '<name>-<number>' format is just a personalized way of expressing the version of that type of model. And the whole 'GPT<number>' category used to be a sort family of models, but now they've just switched it to the actual name of the newer gargantuan LLMs. Then, because the 'GPT<number>' models are now that different thing altogether these days, the newest 'text-embedding' model is just named 'ada-<number>' because it's on that iteration of the 'text-embedding' type of model, adhering to the older principle of naming their models? Not sure, ha. Definitely feels like doing some detective work.
      - simonw 3y ago
        You mean this table here? text-embedding-ada-002 53.3 text-search-davinci-*-001 52.8 text-search-curie-*-001 50.9 text-search-babbage-*-001 50.4 text-search-ada-*-001 49.0 That's not comparing it to the davinci/curie/babbage GPT3 models, it's comparing to the "search-text-*" family. Those were introduced in https://openai.com/blog/introducing-text-and-code-embeddings https://openai.com/blog/introducing-text-and-code-embeddings as the first public release of embeddings models from OpenAI. > We’re releasing three families of embedding models, each tuned to perform well on different functionalities: text similarity, text search, and code search. The models take either text or code as input and return an embedding vector. It's not at all clear to me if there's any relationship between those and the GPT3 davinci/curie/babbage/ada models. My guess is that OpenAI's naming convention back then was "davinci is the best one, then curie, then babbage, then ada".

Their OpenAI benchmark is GPT3 (text-embedding-ada-002), not GPT4.

by e1g 3y ago

Their OpenAI benchmark is GPT3 (text-embedding-ada-002), not GPT4.

"text-embedding-ada-002" isn't GPT3, it's a different kind of model. Embedding models and Large Language Models aren't the same thing.

e1g 3y ago
LLMs and embedding models are certainly different, but it's a useful benchmark to calibrate expectations. OpenAI released text-embedding-ada-002 a year ago, and they describe the ada model as[1] "the original GPT-3 base model [...] capable of very simple tasks, usually the fastest model in the GPT-3 series". It's fair to expect GPT3-level results - not GPT 3.5 and certainly not open-source tiny GPT4 as some might think when they read "rivaling OpenAI". [1] https://platform.openai.com/docs/models/whisper https://platform.openai.com/docs/models/whisper
- minimaxir 3y ago
  When people talked about GPT-3 they always referred to davinci which is the largest model, not ada.
- simonw 3y ago
  No, you're confusing two things here. "text-ada-001" is LLM in the GPT3 family, described as "Capable of very simple tasks, usually the fastest model in the GPT-3 series, and lowest cost" "text-embedding-ada-002" is entirely different - that page describes it as "Our second generation embedding model, text-embedding-ada-002 is a designed to replace the previous 16 first-generation embedding models at a fraction of the cost."
  - minimaxir 3y ago
    tl;dr OpenAI is bad at product naming.
  - e1g 3y ago
    OpenAI doesn't say directly what text-embedding-ada-002 is, but in the release blog post they show that performance is comparable to davinci/curie, which places it firmly in the universe of GPT3. I understand it's not a straight line comparison, but to me it's still a useful mental heuristic about what to expect. [1] https://openai.com/blog/new-and-improved-embedding-model https://openai.com/blog/new-and-improved-embedding-model (see "Model improvements")
    - helloplanets 3y ago
      Reading through that article, the specific Davinci/Curie models they seem to be referring to are called the following: 'text-search-davinci-001', 'text-search-curie-001', 'text-similarity-davinci-001' and 'text-similarity-curie-001'. Are you sure these have anything to do with 'text-davinci-003' or 'text-curie-001'? Will have to agree with everyone here that OpenAI is good at being extremely confusing. It seems like the logic might be something along the lines of the 'text-search' portion being the actual type of the model, while the 'curie-001' / '<name>-<number>' format is just a personalized way of expressing the version of that type of model. And the whole 'GPT<number>' category used to be a sort family of models, but now they've just switched it to the actual name of the newer gargantuan LLMs. Then, because the 'GPT<number>' models are now that different thing altogether these days, the newest 'text-embedding' model is just named 'ada-<number>' because it's on that iteration of the 'text-embedding' type of model, adhering to the older principle of naming their models? Not sure, ha. Definitely feels like doing some detective work.
    - simonw 3y ago
      You mean this table here? text-embedding-ada-002 53.3 text-search-davinci-*-001 52.8 text-search-curie-*-001 50.9 text-search-babbage-*-001 50.4 text-search-ada-*-001 49.0 That's not comparing it to the davinci/curie/babbage GPT3 models, it's comparing to the "search-text-*" family. Those were introduced in https://openai.com/blog/introducing-text-and-code-embeddings https://openai.com/blog/introducing-text-and-code-embeddings as the first public release of embeddings models from OpenAI. > We’re releasing three families of embedding models, each tuned to perform well on different functionalities: text similarity, text search, and code search. The models take either text or code as input and return an embedding vector. It's not at all clear to me if there's any relationship between those and the GPT3 davinci/curie/babbage/ada models. My guess is that OpenAI's naming convention back then was "davinci is the best one, then curie, then babbage, then ada".