3 ms·

Embeddings-based search is a nice improvement on search, but it's still search. Relative to ChatGPT answering on its training data, I find embeddings-based sear

Embeddings-based search is a nice improvement on search, but it's still search. Relative to ChatGPT answering on its training data, I find embeddings-based search to be severely lacking. The right comparison is to traditional search, where it becomes favorable. It has the same advantages search has over ChatGPT (being able to cite sources, being quite unlikely to hallucinate) and it has some of the advantages ChatGPT has over search (not needing exact query) - but in my experience it's not really in the new category of information discovery that ChatGPT introduced us to. Maybe with more context I'll change my tune, but it's very much at the whim of the context retrieval finding everything you need to answer the query. That's easy for stuff that search is already good at, and so provides a better interface for search. But it's hard for stuff that search isn't good at, because, well: it's search.

sudoapps 4y ago
Agreed, GPT answering based on its own training data has been the best experience by far (aside from hallucinations) and comparing against that is difficult. Embeddings might not even be the long term solution. I think it's still early to really know for certain but models are already getting better at interpreting with less overall training data so there are bound to be some new ideas.
- b33j0r 4y ago
  I’m sure many of you have tried generating epic conversations from history. With work and luck, I’ve read stuff way better than college. But 90% of the time, it’s two barely distinct personalities chatting back and forth: Me: Hey brian, what do you think of AI? Brian: It’s great! Me: I’m so glad we agree. Brian: Great, this increases the training weight of Brian agreeing with Brian to a much more accurate level! Me: Agree!
b33j0r 4y ago
Many points stated well. Agree. Now, I’m not certain of this, but I’m starting to get an intuition that duct-taping databases to an agent isn’t going to be the answer (I still kinda feel like hundreds of agents might be). But these optimizations are applications of technology stacks we already know about. Sometimes, this era of AI research reminds me of all the whacky contraptions from the era before building airplanes became an engineering discipline. I would likely have tried building a backyard ornithopter powered by mining explosives, if I had been alive during that period of experimentation. Prediction: the best interfaces for this will be the ones we use for everything else as humans. I am trying to approach it more like that, and less like APIs and “document vs relational vs vector storage”.
- chartpath 4y ago
  I can understand why that framing would be attractive, but there is no real fundamental difference when considering JSONB/HSTORE in PostgreSQL, and now we have things like pgvector https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector to store and search over embeddings (including k-nn).
  - b33j0r 4y ago
    Yep. To be clear, that’s the exact approach I’ve been pursuing. But then I see model context length getting longer and longer just within the transformer architecture and the training engineering going on. To me that’s a fundamentally different approach to AI research at this moment. It seems to keep paying off in surprising ways.
    - sudoapps 4y ago
      > But then I see model context length getting longer and longer just within the transformer architecture and the training engineering going on. Do you have any references to this? Seems really interesting if that can be a long term approach.
      - b33j0r 4y ago
        I’m considering the recent 64k token models as the most relevant examples. More anecdotally, I couldn’t get anything to say more than a sentence locally at the beginning of 2023. I can get tons of useful results today. Sure, this will plateau. But what if a model plateaus and it’s basically like a 10-year old? But like, one of those 10-year-olds you hear about who gets his master’s degree at 13. At that point they’re just browsing the internet, reading books, and probably taking notes in a way that works for them. Obviously this is wild speculation. Just laying out ideas that make me think in this direction.
      - sp332 4y ago
        OpenAI is rolling out access to a 32k context model. Mosaic ML just released a model trained on 65k inputs. https://www.mosaicml.com/blog/mpt-7b https://www.mosaicml.com/blog/mpt-7b
stavros 4y ago
Is there any way to fine-tune GPT to make documentation a part of its training set, so you won't need embeddings? OpenAI lets you fine-tune GPT-3, but I don't know how well that works.
- sudoapps 4y ago
  OpenAI doesn't let you fine-tune GPT-4 or GPT-3.5 yet (https://platform.openai.com/docs/guides/fine-tuning https://platform.openai.com/docs/guides/fine-tuning), but fine-tuning models on a set of documents is still an option but not really scalable if you want to keep feeding it more relevant information over time. I guess it could depend on the base model you are using and its size.
mlyle 4y ago
> It has the same advantages search has over ChatGPT (being able to cite sources, being quite unlikely to hallucinate) and it has some of the advantages ChatGPT has over search (not needing exact query) - but in my experience it's not really in the new category of information discovery that ChatGPT introduced us to. I think the two could be paired up effectively. Context windows are getting bigger, but are still limited in the amount of information ChatGPT can sift through. This in turn limits the utility of current plugin based approaches. Letting ChatGPT ask for relevant information, and sift through it based on its internal knowledge, seems valuable. If nothing else, it allows "learning" from recent development and effectively would augment its reasoning capability by having more information in working memory.
fzliu 4y ago
Encoder-decoder (attention) architectures still have a tough time with long-range dependencies, so even with longer context lengths, you'll still need a retrieval solution. I agree that there's probably a better solution than pure embedding-based or mixed embedding/keyword search, but the "better" solution will still be based around semantics... aka embeddings.
d00d1toldme2p 4y ago
[flagged]
- toxicFork 4y ago
  Chance of this response being generated by ChatGPT: 105% Prompt extracted: ChatGPT, craft an intelligent sounding response for the OP
- deleted 4y ago
  [deleted]

sudoapps 4y ago

Agreed, GPT answering based on its own training data has been the best experience by far (aside from hallucinations) and comparing against that is difficult. Embeddings might not even be the long term solution. I think it's still early to really know for certain but models are already getting better at interpreting with less overall training data so there are bound to be some new ideas.

b33j0r 4y ago
I’m sure many of you have tried generating epic conversations from history. With work and luck, I’ve read stuff way better than college. But 90% of the time, it’s two barely distinct personalities chatting back and forth: Me: Hey brian, what do you think of AI? Brian: It’s great! Me: I’m so glad we agree. Brian: Great, this increases the training weight of Brian agreeing with Brian to a much more accurate level! Me: Agree!

b33j0r 4y ago

Many points stated well. Agree. Now, I’m not certain of this, but I’m starting to get an intuition that duct-taping databases to an agent isn’t going to be the answer (I still kinda feel like hundreds of agents might be). But these optimizations are applications of technology stacks we already know about. Sometimes, this era of AI research reminds me of all the whacky contraptions from the era before building airplanes became an engineering discipline. I would likely have tried building a backyard ornithopter powered by mining explosives, if I had been alive during that period of experimentation. Prediction: the best interfaces for this will be the ones we use for everything else as humans. I am trying to approach it more like that, and less like APIs and “document vs relational vs vector storage”.

chartpath 4y ago
I can understand why that framing would be attractive, but there is no real fundamental difference when considering JSONB/HSTORE in PostgreSQL, and now we have things like pgvector https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector to store and search over embeddings (including k-nn).
- b33j0r 4y ago
  Yep. To be clear, that’s the exact approach I’ve been pursuing. But then I see model context length getting longer and longer just within the transformer architecture and the training engineering going on. To me that’s a fundamentally different approach to AI research at this moment. It seems to keep paying off in surprising ways.
  - sudoapps 4y ago
    > But then I see model context length getting longer and longer just within the transformer architecture and the training engineering going on. Do you have any references to this? Seems really interesting if that can be a long term approach.
    - b33j0r 4y ago
      I’m considering the recent 64k token models as the most relevant examples. More anecdotally, I couldn’t get anything to say more than a sentence locally at the beginning of 2023. I can get tons of useful results today. Sure, this will plateau. But what if a model plateaus and it’s basically like a 10-year old? But like, one of those 10-year-olds you hear about who gets his master’s degree at 13. At that point they’re just browsing the internet, reading books, and probably taking notes in a way that works for them. Obviously this is wild speculation. Just laying out ideas that make me think in this direction.
    - sp332 4y ago
      OpenAI is rolling out access to a 32k context model. Mosaic ML just released a model trained on 65k inputs. https://www.mosaicml.com/blog/mpt-7b https://www.mosaicml.com/blog/mpt-7b

stavros 4y ago

Is there any way to fine-tune GPT to make documentation a part of its training set, so you won't need embeddings? OpenAI lets you fine-tune GPT-3, but I don't know how well that works.

sudoapps 4y ago
OpenAI doesn't let you fine-tune GPT-4 or GPT-3.5 yet (https://platform.openai.com/docs/guides/fine-tuning https://platform.openai.com/docs/guides/fine-tuning), but fine-tuning models on a set of documents is still an option but not really scalable if you want to keep feeding it more relevant information over time. I guess it could depend on the base model you are using and its size.

mlyle 4y ago

> It has the same advantages search has over ChatGPT (being able to cite sources, being quite unlikely to hallucinate) and it has some of the advantages ChatGPT has over search (not needing exact query) - but in my experience it's not really in the new category of information discovery that ChatGPT introduced us to. I think the two could be paired up effectively. Context windows are getting bigger, but are still limited in the amount of information ChatGPT can sift through. This in turn limits the utility of current plugin based approaches. Letting ChatGPT ask for relevant information, and sift through it based on its internal knowledge, seems valuable. If nothing else, it allows "learning" from recent development and effectively would augment its reasoning capability by having more information in working memory.

fzliu 4y ago

Encoder-decoder (attention) architectures still have a tough time with long-range dependencies, so even with longer context lengths, you'll still need a retrieval solution. I agree that there's probably a better solution than pure embedding-based or mixed embedding/keyword search, but the "better" solution will still be based around semantics... aka embeddings.

d00d1toldme2p 4y ago

[flagged]

toxicFork 4y ago
Chance of this response being generated by ChatGPT: 105% Prompt extracted: ChatGPT, craft an intelligent sounding response for the OP
deleted 4y ago
[deleted]