3 ms·
I was wondering if these are fly by night kinda random AI startup type companies offering these services. >we put the claims of two providers, LexisNexis (crea
by duxup 2y ago
I was wondering if these are fly by night kinda random AI startup type companies offering these services.
>we put the claims of two providers, LexisNexis (creator of Lexis+ AI) and Thomson Reuters (creator of Westlaw AI-Assisted Research and Ask Practical Law AI)), to the test
LexisNexis, and Thomson Reuters, those are established big companies offering these services.
- __loam 2y agoI would be interested in how specialist companies like Harvey, which is backed by OpenAI, are doing in this space. They're pretty secretive though.
- janalsncm 2y agoIf I was a law firm, I would not trust my client’s information with Harvey or OpenAI. It honestly doesn’t matter what they promise if my reputation is on the line. No company can promise not to be hacked or leaked.
- deleted 2y ago[deleted]
- tiahura 2y agoBut your firm can guarantee it won’t be hacked? Or broken into. Or social engineered. Or…
- meepmorp 2y agoWhat's more likely - someone targeting your particular law firm or someone targeting OpenAI?
- sjy 2y agoI’d say a small law firm. They don’t have a security team, they’re routinely targeted by phishers, and their data is easier to convert into money than the huge mass of (mostly completely uninteresting) data fed into ChatGPT.
- __loam 2y agoI agree from the perspective that they're probably using the data to do training which is a big big no no in the space, but in terms of hacking, that's true of basically any platform. Certifications like SOCII and Fedramp exist, and even old players like relativity are moving into the cloud. Unless law firms suddenly start wanting to manage their own IT infrastructure, trusting some slice of tech is necessary.
- engineer_22 2y agoThey're established companies with expertise in databases and web GUI. They've got the same talent constraints as everyone else pursuing bespoke LLM implementations.
- danielmarkbruce 2y agoI'd suggest they face more constraints, to the point of it being impossible for them to create something good here: 1) Executives who don't understand LLMs at all and don't really understand the process of building products under massive uncertainty around product quality. 2) Executives who have grown up building products which didn't need insane amounts of continual tuning. 3) Internal process that isn't set up for the types of products mentioned in 1 and 2 4) Entire groups of stakeholders in the same position as executives, putting pressure on the people building these things. To make an analogy - this space is closer to the self driving car project at Google. Zero product risk, the technology is sort of all there, but putting it all together is difficult and it's gonna be ready when it's ready. Pushing something out to meet a schedule aint gonna work.
- deleted 2y ago[deleted]
- hn_throwaway_99 2y agoI'd be interested to see other examples of Lexis+ AI failures, because I thought the example answer shown in the article appeared to be the "least wrong" one given that the answer was correct until Dobbs was decided less than 2 years ago. I'm not in the LLM training space, but I'd also think that that kind of failure would be the easiest to fix, i.e. instead of starting out with "Currently...", start out with "As of <date>..."
- vharuck 2y agoIs there actually an effective way to handle queries with RAG where time periods are relevant? I made a proof-of-concept RAG for documents on a government website shortly after GPT-3.5 came out and remember this being a big problem. The most glaring wrong answer was "Who is currently the governor?" It answered with the previous governor, likely because he was listed as such in 8 years of documents versus the 2 years at the time for the current governor.
- rfw300 2y agoThere's more examples on page 18 of the paper: https://dho.stanford.edu/wp-content/uploads/Legal_RAG_Hallucinations.pdf https://dho.stanford.edu/wp-content/uploads/Legal_RAG_Halluc...
- hn_throwaway_99 2y agoThanks very much for linking that. This makes me think that legal support is actually one of the worst possible uses of LLM-based AI (at least as implemented here), primarily because so much of the source material is directly contradictory, e.g. a legislature passes a law which is subsequently overturned by the courts, or decisions in lower courts are reversed by higher courts, or higher courts reverse themselves over time. It feels like you'd absolutely have to annotate all the source material in some way to say whether it was still controlling law/precedent.
- rfw300 2y agoYour instinct is correct. The major legal research providers (Thomson Reuters and LexisNexis) both provide “citators”, which are human annotations of which cases and statutes have been overruled, upheld, criticized, etc. One of the issues the paper describes is the fairly ham-handed way this gets integrated into these systems, causing even more trouble.