3 ms·

Some caution here. Not everything needs to go into a RAG pipeline (Eg: a database table would not necessarily need to be embedded, but its schema should be.). T

Some caution here. Not everything needs to go into a RAG pipeline (Eg: a database table would not necessarily need to be embedded, but its schema should be.). There would be a lot of repetitions, lots of junk and useless data, and numerical data and parsing through that would be a pain. Then comes how the users would behave. You need a longer string to get accurate results. Most non tech users would rather write shorter strings and expect technology to read their mind. (it's a human issue and not tech issue) A simpler way here is just train the model unsupervised so all the knowledge is there in the model, and instruction tune it on the use-cases you want. Simpler from human effort perspective. Somewhat costly though the cost of storing that many vectors would be more than training the model itself. Everything else requires a lot of custom effort. Knowledge graph augmentation is probably the next step in the hype cycle, but it does not solve the fundamental human problem of writing fewer letters. (Training solves as changing 1-2 keywords do the trick if the generic string does not get the answer. See how Chatgpt changes answers if you tweak your prompt a bit). In a way RAG is an engineering solution to what is basically a data problem. It works for many cases, but when it does not, people will have to solve it via data science. > Wow so RAG is basically a toy for demos and low effort MVPs I would not say it's for demos or low effort MVPs. Many companies wont have that amount of data. You can also segregate it by teams. Eg: customer support has one, sales has one, product has one. Then, a golden use case is for parsing user docs. We created one for GST queries in India that works quite well.[1]. It's a search engine, but points to right docs at the source when you ask about any clause. Useful for CAs only and addresses a very narrow use case.(it's a market need as the notifications are published in PDF format and not indexed by Google) [1]:https://clioapp.ai/gst-search https://clioapp.ai/gst-search

throwup238 3y ago
"Toy" is the wrong word to describe it but it seems like another order of magnitude or two increase in context size will solve all their problems. On the other hand I've got a terabyte of text extracted from LibGen - let's say I can ignore the half that is fiction and I can dedupe the rest further by 80% - that's still 100gb. On top of that I've got 300gb of text extracted from court documents and that's just from California! I haven't even downloaded the federal dump yet, let alone the other 49 states. Even if I limited myself to just the US Code and Federal Code of Regulations, that's hundreds of millions of tokens of very dense text. Embedding based RAG has been pretty much useless in each of these cases but maybe I just suck at implementing the retrieval part. What's the data size on the GST search engine? Do you have any examples of more complex queries? The only thing that has even been remotely useful to tackling the kind of questions I want to ask of my data sources is having the LLM generate search queries, navigate the knowledge graph, and rank the relevance of retrieved snippets but that often takes dozens if not hundreds of LLM calls which is incredibly slow and expensive.
- rolisz 3y ago
  What are you trying to achieve with that dataset from LibGen? I kinda expect that GPT4 was trained on the data that is available on LibGen
  - throwup238 3y ago
    I need it to be able to cite answers, explore surrounding context, and while it might have been trained on Libgen, it doesn't mean it "internalized" all the data, let alone enough to be useful.
- fennecbutt 3y ago
  As a human would you read 100GB of data all at once? Or would you read it bit by bit, taking notes and summarising as you went along. Then compiling your notes/summaries into a final report? Because I don't see why we expect these models to be so superhuman when a 100K context would already be considered superhuman memory. Imagine me regurgitating 100k tokens worth of dialogue at you and expecting you to take into account every thing I said. I know I couldn't do it, ha ha.
  - throwup238 3y ago
    As a human would you do tens of billions of multiplies and additions per second? Store tens of thousands of books on something the size of a finger nail and recall them with perfect fidelity every time? Communicate with another human via optical signals using thousand mile long optical fiber across the entire Pacific ocean? Eat electricity instead of food? Project images from your eyes? Can you stick an audio cable in your butt to power speakers? I'm talking about computers, not humans.
    - fennecbutt 3y ago
      I mean I wouldn't put it past myself to try the audio cable in the butt trick, bet it'd feel great
- MilStdJunkie 3y ago

throwup238 3y ago

"Toy" is the wrong word to describe it but it seems like another order of magnitude or two increase in context size will solve all their problems. On the other hand I've got a terabyte of text extracted from LibGen - let's say I can ignore the half that is fiction and I can dedupe the rest further by 80% - that's still 100gb. On top of that I've got 300gb of text extracted from court documents and that's just from California! I haven't even downloaded the federal dump yet, let alone the other 49 states. Even if I limited myself to just the US Code and Federal Code of Regulations, that's hundreds of millions of tokens of very dense text. Embedding based RAG has been pretty much useless in each of these cases but maybe I just suck at implementing the retrieval part. What's the data size on the GST search engine? Do you have any examples of more complex queries? The only thing that has even been remotely useful to tackling the kind of questions I want to ask of my data sources is having the LLM generate search queries, navigate the knowledge graph, and rank the relevance of retrieved snippets but that often takes dozens if not hundreds of LLM calls which is incredibly slow and expensive.

rolisz 3y ago
What are you trying to achieve with that dataset from LibGen? I kinda expect that GPT4 was trained on the data that is available on LibGen
- throwup238 3y ago
  I need it to be able to cite answers, explore surrounding context, and while it might have been trained on Libgen, it doesn't mean it "internalized" all the data, let alone enough to be useful.
fennecbutt 3y ago
As a human would you read 100GB of data all at once? Or would you read it bit by bit, taking notes and summarising as you went along. Then compiling your notes/summaries into a final report? Because I don't see why we expect these models to be so superhuman when a 100K context would already be considered superhuman memory. Imagine me regurgitating 100k tokens worth of dialogue at you and expecting you to take into account every thing I said. I know I couldn't do it, ha ha.
- throwup238 3y ago
  As a human would you do tens of billions of multiplies and additions per second? Store tens of thousands of books on something the size of a finger nail and recall them with perfect fidelity every time? Communicate with another human via optical signals using thousand mile long optical fiber across the entire Pacific ocean? Eat electricity instead of food? Project images from your eyes? Can you stick an audio cable in your butt to power speakers? I'm talking about computers, not humans.
  - fennecbutt 3y ago
    I mean I wouldn't put it past myself to try the audio cable in the butt trick, bet it'd feel great
MilStdJunkie 3y ago