5 ms·
> What will replace Internet data for training? Curated synthetic datasets? Perhaps a different take at this could be: if I wanted to train "state law" LLM tha
by menaerus 2y ago
> What will replace Internet data for training? Curated synthetic datasets?
Perhaps a different take at this could be: if I wanted to train "state law" LLM that is exceedingly good in interpreting state law, what are the obstacles to download all the law and regulations material for given state that will allow me to train LLM such that it becomes 95th percentile of all law trainees and lawyers.
In that case, and my point is, that we already don't need an "Internet". We just need a sufficiently sized and curated domain-specific dataset and the result we can get is already scary. "State law" LLM was just an example but the same logic applies to basically any other domain - want a domain-specific (LLM) expert? Train it.
- pas 2y agoyou need context for the dry statutes sure, you download all the legal arguments, and hope that putting all this on top of a general LLM which has enough context to deal with usual human, American, contemporary stuff the argument is that it's not really enough for the next jump (as it would need "exponentially" more data) as far as a I understand
- menaerus 2y agoI don't understand the limitation, e.g. how much data do you need to train the "law state" specific LLM that doesn't know anything else but that? Such LLM does not need to have 400B of parameters since it's not a general knowledge LLM but perhaps I'm wrong on this (?). So my point rather is that it may very well be, let's take for example, a 30B parameters LLM which in turn means that we might have just enough data to train it. Larger contexts in smaller models are a solved problem.
- petesergeant 2y ago> how much data do you need to train the "law state" specific LLM that doesn't know anything else but that? Law doesn’t exist in a vacuum. You can’t have a useful LLM for state law that doesn’t have an exceptional grounding in real world objects of mechanics. You could force a bright young child to memorize a large text, but without a strong general model of the world, they’re just regurgitating words rather than able to reason about it.
- menaerus 2y agoCounter-argument: code does not exist in vacuum yet we have small and mid-sized LLMs that can already output reasonable code.
- petesergeant 2y agoGenerally they’ve been distilled from much larger models, but also, code is a much smaller domain than the law.
- noirbot 2y agoCode is both much smaller as a domain and less prone to the chaos of human interpretation. There are many factors that go into why a given civil or criminal case in court turns out how it does, and often the biggest one is not "was it legal". Giving a computer access to the full written history of cases doesn't give you any of the context of why those cases turned out. A judge or jury isn't going to include in the written record that they just really didn't like one of the lawyers. Or that the case settled because one of the parties just couldn't afford to keep going. Or that one party or the other destroyed/withheld evidence. Generally speaking, your compiler won't just decide not to work as expected. Tons of legal decisions don't actually follow the law as written. Or even the precedent set by other courts. And that's even assuming the law and precedent are remotely clear in the first place.
- zozbot234 2y agoA model that's trained on legal decisions can still be used to explore these questions, though. The model may end up being uncertain about which way the case will go, or even more strikingly, it may be confident about the outcome of a case that then is decided differently, and you can try and figure out what's going on with such cases.
- sharih 2y agolegal reasoning involves applying facts to the law, and it needs knowledge of the world. the expertise of a professional is in picking the right/winning path based on their study of the law, the facts and their real world training. money is in codifying that to teach models to do the same
- theptip 2y agoFor a “legal LLM” you need three things: general IQ / common sense at a substantially higher level than current, understanding of the specific rules, and hallucination-free recall of the relevant legal facts/cases. I think it’s reasonable to assume you can get 2/3 with a small corpus IF you have an IQ 150 AGI. Empirically the current known method for increasing IQ is to make the model bigger. Part of what you’re getting at is possible though, once you have the big model you can distill it down to a smaller number of parameters without losing much capability in your chosen narrow domain. So you forget physics and sports but remain good at law. That doesn’t help you with improving the capability frontier though.
- pas 2y agoAnd then your Juris R. Genius gets a new case about two Red Socks fans getting into a fight and without missing a beat starts blabbering about how overdosing on too much red pigments from the undergarments caused their rage!
- theptip 2y agoYeah, I think for the highest-value activities (eg legal advice) you expect to run the full frontier model. But maybe you want to run a smaller one locally on your iPhone for privacy and accept the capability loss.
- losvedir 2y agoThat's kind of going in a different direction. The big picture is that LLMs have until this point gotten better and better from larger datasets alone. See "The Bitter Lesson". But now we're running out of datasets and so the only way we know of to improve models' reasoning abilities and everything, is coming to an end. You're talking about fine tuning, which yes is a technique that's being used and explored in different domains, but my understanding is it's not a very good way for models to acquire knowledge. Instead larger context windows and RAG works better for something like case law. Fine tuning works for things like giving models a certain "voice" in how they produce text, and general alignment things. At least that's my understanding as an interested but not totally involved follower of this stuff.
- yeahwhatever10 2y agoThe problem remains the size of the dataset. You aren't going to get large enough datasets in these specific domains.
- sharih 2y agothe big frontier models already have all laws, regulations and cases memorized/trained on given they are public. the real advancement is in experts codifying their expertise/reasoning for models to learn from. legal is no different from other fields in this.
- menaerus 2y agoSo, fine-tuning the model to the law of the exact country. Or fine-tuning the model to the problem space of the exact codebase/product. You hire 10 law experts instead of 100. Or you hire 10 programmers instead of 100. Expertise is becoming a commodity I'm afraid.