4 ms·

Using AI to search for case law: it makes cases up – it does

2 points by raattgift 3y ago

h2odragon 3y ago
playing with one of the "llamafile" llms last night, one of them told me "there's been 3 cases impacting the 3rd amendment" and then listed 4 made up references. The "Law is copyrwrit by us" jerks and the "we could use better search" users of that data are in for a fun confrontation very shortly here. There's masses of legal data that isn't copyrighted; the Federal register and Congressional Record are available in lovely formats and the data goes back a ways. I'm waiting for people to start experimenting with the text mining and concordance kind of things LLM techniques can do on those data sets. There's already a market for that kind of thing, but those established systems are dinosaurs that will crumble with a push. cf "Managing Gigabytes" and trigraph indexing. Nifty tricks but decidedly less than optimal.
- dragonwriter 3y ago
  > I'm waiting for people to start experimenting with the text mining and concordance kind of things LLM techniques can do on those data sets. There's already a market for that kind of thing, but those established systems are dinosaurs that will crumble with a push. Those established systems have a wealth of expert human annotations and related information that is either not readily publicly available and copyright protected (including their own first-party data), including masses of licensed texts, as well as the ability to enhance what they have with usage data from their extensive network of existing professional and academic users. Yes, LLMs will transform the space, but LLMs tied into the systems and proprietary data sets of the existing incumbents are likely to be the professional standard and way better than what is publicly available in the near term, unless the incumbents really fumble the ball. That said, sure, public LLM based systems tying into public, noncopyrighted data can make what is available to the public without expensive subscriptions a lot better than it is today.

2 points by raattgift 3y ago

playing with one of the "llamafile" llms last night, one of them told me "there's been 3 cases impacting the 3rd amendment" and then listed 4 made up references. The "Law is copyrwrit by us" jerks and the "we could use better search" users of that data are in for a fun confrontation very shortly here. There's masses of legal data that isn't copyrighted; the Federal register and Congressional Record are available in lovely formats and the data goes back a ways. I'm waiting for people to start experimenting with the text mining and concordance kind of things LLM techniques can do on those data sets. There's already a market for that kind of thing, but those established systems are dinosaurs that will crumble with a push. cf "Managing Gigabytes" and trigraph indexing. Nifty tricks but decidedly less than optimal.

dragonwriter 3y ago
> I'm waiting for people to start experimenting with the text mining and concordance kind of things LLM techniques can do on those data sets. There's already a market for that kind of thing, but those established systems are dinosaurs that will crumble with a push. Those established systems have a wealth of expert human annotations and related information that is either not readily publicly available and copyright protected (including their own first-party data), including masses of licensed texts, as well as the ability to enhance what they have with usage data from their extensive network of existing professional and academic users. Yes, LLMs will transform the space, but LLMs tied into the systems and proprietary data sets of the existing incumbents are likely to be the professional standard and way better than what is publicly available in the near term, unless the incumbents really fumble the ball. That said, sure, public LLM based systems tying into public, noncopyrighted data can make what is available to the public without expensive subscriptions a lot better than it is today.