3 ms·

Many points stated well. Agree. Now, I’m not certain of this, but I’m starting to get an intuition that duct-taping databases to an agent isn’t going to be the

by b33j0r 3y ago

Many points stated well. Agree. Now, I’m not certain of this, but I’m starting to get an intuition that duct-taping databases to an agent isn’t going to be the answer (I still kinda feel like hundreds of agents might be). But these optimizations are applications of technology stacks we already know about. Sometimes, this era of AI research reminds me of all the whacky contraptions from the era before building airplanes became an engineering discipline. I would likely have tried building a backyard ornithopter powered by mining explosives, if I had been alive during that period of experimentation. Prediction: the best interfaces for this will be the ones we use for everything else as humans. I am trying to approach it more like that, and less like APIs and “document vs relational vs vector storage”.

chartpath 3y ago
I can understand why that framing would be attractive, but there is no real fundamental difference when considering JSONB/HSTORE in PostgreSQL, and now we have things like pgvector https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector to store and search over embeddings (including k-nn).
- b33j0r 3y ago
  Yep. To be clear, that’s the exact approach I’ve been pursuing. But then I see model context length getting longer and longer just within the transformer architecture and the training engineering going on. To me that’s a fundamentally different approach to AI research at this moment. It seems to keep paying off in surprising ways.
  - sudoapps 3y ago
    > But then I see model context length getting longer and longer just within the transformer architecture and the training engineering going on. Do you have any references to this? Seems really interesting if that can be a long term approach.
    - b33j0r 3y ago
      I’m considering the recent 64k token models as the most relevant examples. More anecdotally, I couldn’t get anything to say more than a sentence locally at the beginning of 2023. I can get tons of useful results today. Sure, this will plateau. But what if a model plateaus and it’s basically like a 10-year old? But like, one of those 10-year-olds you hear about who gets his master’s degree at 13. At that point they’re just browsing the internet, reading books, and probably taking notes in a way that works for them. Obviously this is wild speculation. Just laying out ideas that make me think in this direction.
    - sp332 3y ago
      OpenAI is rolling out access to a 32k context model. Mosaic ML just released a model trained on 65k inputs. https://www.mosaicml.com/blog/mpt-7b https://www.mosaicml.com/blog/mpt-7b
sebzim4500 3y ago
My intuition is that it would work much better if the model could choose what to search for with something like langchain. The problem is that we don't know how to train such a system properly, we mainly do supervised finetuning on human examples of using the tools but this is fundamentally a reinforcement learning problem (RL is just hard).
- b33j0r 3y ago
  That’s what I have been up to as well. I just think we’re so traumatized from the past 15 years of software engineering (kidding) that we’re stuck thinking “hey, let’s add memcached, that solved many problems like this before!” Natural language is weird. It’s linear in the sense that you can’t take back what you said. However, subjects and verbs and parentheticals and objects are like electron clouds. We only know statistically where it’s going to be. And damn, you’re reading this now, somehow, anyway. Seems like the transformer model really is a new technology. But it’s just that we can actually run useful NLP neural networks now, which has opened the floodgate of innovation in various subfields. (Personally bothers me because absolutely no one took it seriously when I was starting my career! Lol. I had a meeting with my AI professor where they told me ANN’s were nonsense even if we had computers 1000x more powerful. Prune the search tree! That was not very long ago).
kordlessagain 3y ago
Vector search with move tos and move aways based on feedback is much more than attaching a database…

by b33j0r 3y ago

chartpath 3y ago

I can understand why that framing would be attractive, but there is no real fundamental difference when considering JSONB/HSTORE in PostgreSQL, and now we have things like pgvector https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector to store and search over embeddings (including k-nn).

b33j0r 3y ago
Yep. To be clear, that’s the exact approach I’ve been pursuing. But then I see model context length getting longer and longer just within the transformer architecture and the training engineering going on. To me that’s a fundamentally different approach to AI research at this moment. It seems to keep paying off in surprising ways.
- sudoapps 3y ago
  > But then I see model context length getting longer and longer just within the transformer architecture and the training engineering going on. Do you have any references to this? Seems really interesting if that can be a long term approach.
  - b33j0r 3y ago
    I’m considering the recent 64k token models as the most relevant examples. More anecdotally, I couldn’t get anything to say more than a sentence locally at the beginning of 2023. I can get tons of useful results today. Sure, this will plateau. But what if a model plateaus and it’s basically like a 10-year old? But like, one of those 10-year-olds you hear about who gets his master’s degree at 13. At that point they’re just browsing the internet, reading books, and probably taking notes in a way that works for them. Obviously this is wild speculation. Just laying out ideas that make me think in this direction.
  - sp332 3y ago
    OpenAI is rolling out access to a 32k context model. Mosaic ML just released a model trained on 65k inputs. https://www.mosaicml.com/blog/mpt-7b https://www.mosaicml.com/blog/mpt-7b

sebzim4500 3y ago

My intuition is that it would work much better if the model could choose what to search for with something like langchain. The problem is that we don't know how to train such a system properly, we mainly do supervised finetuning on human examples of using the tools but this is fundamentally a reinforcement learning problem (RL is just hard).

b33j0r 3y ago
That’s what I have been up to as well. I just think we’re so traumatized from the past 15 years of software engineering (kidding) that we’re stuck thinking “hey, let’s add memcached, that solved many problems like this before!” Natural language is weird. It’s linear in the sense that you can’t take back what you said. However, subjects and verbs and parentheticals and objects are like electron clouds. We only know statistically where it’s going to be. And damn, you’re reading this now, somehow, anyway. Seems like the transformer model really is a new technology. But it’s just that we can actually run useful NLP neural networks now, which has opened the floodgate of innovation in various subfields. (Personally bothers me because absolutely no one took it seriously when I was starting my career! Lol. I had a meeting with my AI professor where they told me ANN’s were nonsense even if we had computers 1000x more powerful. Prune the search tree! That was not very long ago).

kordlessagain 3y ago

Vector search with move tos and move aways based on feedback is much more than attaching a database…