3 ms·

"Toy" is the wrong word to describe it but it seems like another order of magnitude or two increase in context size will solve all their problems. On the other

"Toy" is the wrong word to describe it but it seems like another order of magnitude or two increase in context size will solve all their problems. On the other hand I've got a terabyte of text extracted from LibGen - let's say I can ignore the half that is fiction and I can dedupe the rest further by 80% - that's still 100gb. On top of that I've got 300gb of text extracted from court documents and that's just from California! I haven't even downloaded the federal dump yet, let alone the other 49 states. Even if I limited myself to just the US Code and Federal Code of Regulations, that's hundreds of millions of tokens of very dense text. Embedding based RAG has been pretty much useless in each of these cases but maybe I just suck at implementing the retrieval part. What's the data size on the GST search engine? Do you have any examples of more complex queries? The only thing that has even been remotely useful to tackling the kind of questions I want to ask of my data sources is having the LLM generate search queries, navigate the knowledge graph, and rank the relevance of retrieved snippets but that often takes dozens if not hundreds of LLM calls which is incredibly slow and expensive.

rolisz 3y ago
What are you trying to achieve with that dataset from LibGen? I kinda expect that GPT4 was trained on the data that is available on LibGen
- throwup238 3y ago
  I need it to be able to cite answers, explore surrounding context, and while it might have been trained on Libgen, it doesn't mean it "internalized" all the data, let alone enough to be useful.
fennecbutt 3y ago
As a human would you read 100GB of data all at once? Or would you read it bit by bit, taking notes and summarising as you went along. Then compiling your notes/summaries into a final report? Because I don't see why we expect these models to be so superhuman when a 100K context would already be considered superhuman memory. Imagine me regurgitating 100k tokens worth of dialogue at you and expecting you to take into account every thing I said. I know I couldn't do it, ha ha.
- throwup238 3y ago
  As a human would you do tens of billions of multiplies and additions per second? Store tens of thousands of books on something the size of a finger nail and recall them with perfect fidelity every time? Communicate with another human via optical signals using thousand mile long optical fiber across the entire Pacific ocean? Eat electricity instead of food? Project images from your eyes? Can you stick an audio cable in your butt to power speakers? I'm talking about computers, not humans.
  - fennecbutt 3y ago
    I mean I wouldn't put it past myself to try the audio cable in the butt trick, bet it'd feel great
MilStdJunkie 3y ago
Forgive a relative layman chiming in, but isn't legal corpus already pre-chunked in various forms, like section/para/etc., i.e. 18 U.S. Code § 371? It seems that you could slice up the data, RAG from the slices, then connect something like Mixtral's so-called "mixture of experts” (MoE, i.e. 8x7b) for combinations. Word of warning: we've done the "slicing" thing with aerospace data LLMing - we had a similar problem to yours, so we just made RAGs for each functional system (Fuel, Engine, Model for maintenance logs per 14 CFR 43, 91, etc) based on some simple filename filters - but not the MoE thing. Sigh. Stoppage was not due to failure but due to . . let's say, lack of interest. No one wants to solve the problem in-house, but at the same time, no one's allowed to use any cloud-based LLM solutions off the shelf. Far easier to sit on one's hands, and wait for the program to yell at you.

rolisz 3y ago

What are you trying to achieve with that dataset from LibGen? I kinda expect that GPT4 was trained on the data that is available on LibGen

throwup238 3y ago
I need it to be able to cite answers, explore surrounding context, and while it might have been trained on Libgen, it doesn't mean it "internalized" all the data, let alone enough to be useful.

fennecbutt 3y ago

As a human would you read 100GB of data all at once? Or would you read it bit by bit, taking notes and summarising as you went along. Then compiling your notes/summaries into a final report? Because I don't see why we expect these models to be so superhuman when a 100K context would already be considered superhuman memory. Imagine me regurgitating 100k tokens worth of dialogue at you and expecting you to take into account every thing I said. I know I couldn't do it, ha ha.

throwup238 3y ago
As a human would you do tens of billions of multiplies and additions per second? Store tens of thousands of books on something the size of a finger nail and recall them with perfect fidelity every time? Communicate with another human via optical signals using thousand mile long optical fiber across the entire Pacific ocean? Eat electricity instead of food? Project images from your eyes? Can you stick an audio cable in your butt to power speakers? I'm talking about computers, not humans.
- fennecbutt 3y ago
  I mean I wouldn't put it past myself to try the audio cable in the butt trick, bet it'd feel great

MilStdJunkie 3y ago

Forgive a relative layman chiming in, but isn't legal corpus already pre-chunked in various forms, like section/para/etc., i.e. 18 U.S. Code § 371? It seems that you could slice up the data, RAG from the slices, then connect something like Mixtral's so-called "mixture of experts” (MoE, i.e. 8x7b) for combinations. Word of warning: we've done the "slicing" thing with aerospace data LLMing - we had a similar problem to yours, so we just made RAGs for each functional system (Fuel, Engine, Model for maintenance logs per 14 CFR 43, 91, etc) based on some simple filename filters - but not the MoE thing. Sigh. Stoppage was not due to failure but due to . . let's say, lack of interest. No one wants to solve the problem in-house, but at the same time, no one's allowed to use any cloud-based LLM solutions off the shelf. Far easier to sit on one's hands, and wait for the program to yell at you.