10 ms·

AI outperforms law professors in Stanford Law study

• 34981t 4mo ago
  He is basically an AI professor for law. This study just confirms his existence: https://juliannyarko.com/ https://juliannyarko.com/ Stanford and its donors of course want to replace anyone but its administrators, so they cheer on such anti-intellectual nonsense.
  • signatoremo 4mo ago
    This is the state of HN. Created new account. Accused without evidence. Emotional clickbait.
    • vessenes 4mo ago
      I vibe coded hn10k earlier this year. You could choose to see pages with comments only started by 1k+, 10k+ or 100k+ karma contributors. I'm too lazy to keep it up, but I found 1k and 10k both to be better experiences than "vanilla".
• king_zee 4mo ago
  I think there will be a market for firms that aggressively market themselves as non-AI, and then as more people turn towards that human connection we'll go full circle
  • citizenpaul 4mo ago
    If you want human connection the legal system is not where you are going to find it, period. I don't think there will be any such market for "non ai" law. If I'm involved with the legal system I just want out as quick as possible as cheap as possible.
    • applfanboysbgon 4mo ago
      Bad legal advice will keep you dealing with the legal system for much longer and at much greater cost. Something being cheap and quick upfront doesn't mean it will be cheap and quick by the end of the process.
      • Esophagus4 4mo ago
        But isn’t this study saying that the legal advice could actually be better with AI? A bit of extrapolation from the study, but not a crazy stretch.
        • applfanboysbgon 4mo ago
          Maybe, although I would be extremely hesitant to extrapolate from this one study and trust my legal life to an LLM. One thing that's worth noting, though, is that regardless of the quality of objective legal advice in the abstract, for a lot of smaller scale stuff the human connection actually is literally what is important. There are ambiguities in the law, which are not resolved deterministically but rather at the individual discretion of judges. Your lawyer, if they're any good at their job, knows the local judges and how they're likely to rule for given circumstances, which can influence their legal advice to you specifically.
          • Esophagus4 4mo ago
            Fair. But I could also see a world where that, too, is fed to models for hyper-local results. Could be a way off, but I could see it.
      • zuzululu 4mo ago
        I think you are ignoring that there are bad lawyers and they give bad legal advice too Even the good ones will not step above and beyond what they are paid to do but an AI ? it will and can go above and beyond
      • citizenpaul 4mo ago
  • rayiner 4mo ago
    Nobody wants to pay their lawyers more than they have to. There will be a huge market for firms that can use AI to avoid charging clients for $1,000/hour junior associates.
  • zuzululu 4mo ago
    that worked out for artists and translators right ?
• fgh_ask 4mo ago
  [flagged]
  • maxbond 4mo ago
    Just so you know, I have nothing to do with Stanford, but I am flagging this as conspiratorial nonsense. So when you comment is flagged, I just want you to know that it doesn't confirm your belief, it's just that this comment harms discussion and so must be removed.
    • hoppyhoppy2 4mo ago
      >Don't feed egregious comments by replying; flag them instead. If you flag, please don't also comment that you did. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
      • maxbond 4mo ago
        Yes, mea culpa. Occasionally I break that rule on my own judgement. Feel free to flag my comment. (I think it's important to disconfirm conspiracy theories.)
    • thin_carapace 4mo ago
      for what it's worth I have no idea why it would be nonsense to question institutional motivations especially in the context of an academic article that could easily be corporate propaganda, I also think that shutting conversations down is much more harmful than discussing topics that are potentially harmful
      • maxbond 4mo ago
        Completely unevidenced conspiracy theories can only harm the discussion. The only possible benefit is to disconfirm conspiracy theories and discourage paranoid thinking. The odds that Standford as an institution are astroturfing on HN round down to 0. What they're almost certainly observing is that these critical comments are being flagged as inappropriate. People make inappropriate comments that happen to contain criticism all the time, and I frequently see people edit them to declare that they were flagged because the group they're criticizing is astroturfing. It's virtually never the case. I've never seen it happen. But to be clear I am completely ambivalent on Stanford and if you want to criticize them, more power to you.
• wilg 4mo ago
  > In a blind evaluation of nearly 3,000 anonymized comparisons, professors rated AI responses significantly higher than answers written by other professors, with AI winning 75% of head-to-head matchups. 75% win rate seems pretty good! Paper link: https://law.stanford.edu/wp-content/uploads/2026/06/salinas_et_al.pdf https://law.stanford.edu/wp-content/uploads/2026/06/salinas_...
  • causal 4mo ago
    I wonder to what degree the AI was just better at communicating. My experience with attorneys is that they are often some of the worst writers.
    • applicative 4mo ago
      The writing is always fluid and grammatically flawless. This carries much more weight with us than we believe. I know the illusion well from decades of grading college papers. Many of the highest quality students use English as a second language, and I know this, but an American well trained in writing, grammar, spelling always gives an impression of superiority. (Being well trained in writing, grammar, spelling etc is of course high merit, which is how the illusion forms - it is basically an illusion of global 'intelligence')
  • jshier 4mo ago
    I do wish they'd used some more objective criteria. Simply being preferable one of the things LLMs have trained for since the beginning, hence its sycophantic nature.
    • wilg 4mo ago
      What criteria would you use for judging legal arguments?
      • mylifeandtimes 4mo ago
        maybe seeing if the case law it cited was real or imagined? Just one idea, IANAL
• causal 4mo ago
  As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.
  • RataNova 4mo ago
    I think that's the right intuition. Legal AI feels especially dangerous because the output can look competent while hiding jurisdiction-specific footguns
  • thewebguyd 4mo ago
    I think this is probably true for most skilled professions. AI is best used in the hands of folks already knowledgeable in the skills/professions they are using it for. I liken it to me googling things as a sysadmin vs. Jane from accounting doing it. The non-tech end user is far more likely to make the problem worse, or install something sketchy from the ad riddled results than I am, or one of my help desk employees are. I wouldn't trust myself to draft an important legal document using AI without the advice of a lawyer, much like I wouldn't really want to rely on my lawyer to use AI to write code for me.
    • ChrisMarshallNY 4mo ago
      > I wouldn't really want to rely on my lawyer to use AI to write code for me. Yet that is exactly what a lot of C-Suiters (many of whom are lawyers), are doing.
      • xiaoyu2006 4mo ago
        Vice versa there is also a lot of irresponsible programmers doing stupid things with ai. Irresponsible people stay irresponsible, AI just make them more productive at being irresponsible.
        • 4mo ago
• deleted 4mo ago
  [deleted]
• steele 4mo ago
  [flagged]
• jimbokun 4mo ago
  [flagged]
  • deleted 4mo ago
    [deleted]
  • jatora 4mo ago
    definitely not needed if you're in the middle-man slime trades (law)
    • jimbokun 4mo ago
      In an advanced economy everyone’s the middle man for something. We’re not self sustaining agrarian farmers anymore.
    • zuzululu 4mo ago
      what do you think software devs do all day
      • jatora 4mo ago
        create value and utility without purposely gatekeeping and hamstringing society
  • Waterluvian 4mo ago
    the memes were nice tho
• Esophagus4 4mo ago
  Yeah this could be interesting. A lot of the spotlight has been on “law firm stuff” like demand letters and writing contracts… But imagine if a dev team didn’t have to go engineer -> product manager -> legal team to get a question answered on local data retention requirements. You could ship that much faster.
  • ares623 4mo ago
    Would you take responsibility for missing details about local data retention requirements?
    • zuzululu 4mo ago
      honestly if you just avoid EU and China you can get away with anything
      • jedberg 4mo ago
        California too.
        • applfanboysbgon 4mo ago
          And with those three places listed you've ruled out literally 40% of the world economy. Great, you can ship your product in bumfuck Nebraska.
    • Esophagus4 4mo ago
      Yes. If the only purpose of asking a lawyer is transferring risk (aka cover your ass) while getting the same advice as an LLM, that’s slowing down delivery for purely bureaucratic reasons. I’ve seen that mentality at big companies where everyone is scared to stick their neck out and be accountable for a decision. And nothing gets done. Drives me crazy. But the people who move up are the people who take ownership and get shit done (and are right a lot). (BTW, I have been at companies that were sued by regulators. They never really punish the individual(s) who were in the room when the decision is made. So your worry is kind of misplaced.)
• homeonthemtn 4mo ago
  Personally I think this is very good. One of the hardest things out there is maintaining a society in the face of changing times and it's because law is dense and slow. I think, in the right hands, this could be huge.
  • wholinator2 4mo ago
    It turns out everybody has at least one right hand, even the people we trust the least.
• aetq51 4mo ago
  [flagged]
  • ares623 4mo ago
    Running out of IPO juice. Each bump is less effective and lasts shorter.
  • rfw300 4mo ago
    A law professor studying AI has an affiliation with the center at their university that studies applications of AI? Scandalous!
  • wilg 4mo ago
    You're suspicious that the person doing academic research on how AI applies to law has a job related to research on law and AI?
    • runarberg 4mo ago
      You are not? It is at least worth investigating how much this professor benefits from AI companies. In fact this is HN. Let me come back to you in about 10 minutes. EDIT: 10 min later. I give up. I tried to find who is funding HAI, and came empty handed, usually you can see that in their yearly reports, but no such luck for me. I know Google and Bill Gates are big donors, so take that as you will.
  • dang 4mo ago
    Would you please stop creating accounts to post this?
• bko 4mo ago
  Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field. I'm getting more convinced. I mean, sure it makes dumb mistakes sometimes but its a particular set of self serving mistakes, commenting out tests in order to pass. We obv don't want this behavior but I wouldn't say it's dumb. It'll be like the Turing test, which we just blew past years ago and no one cared. After all the hand-wringing about sentience and rights of the AI if it passes the Turing test, and now we just have AI bots running 24/7 writing slop. How does everyone else feel?
  • 12AHg 4mo ago
    [flagged]
    • futuraperdita 4mo ago
      I’m not an AI stan by any means and certainly no fan of Andreessen, but using the term “clanker” immediately biases your statement and can discredit what is a well-referenced or well-meaning comment.
  • paulmist 4mo ago
    Knowing the question is half of the answer. LLMs are great at scoping your context and answering precisely what you asked; it's also why they go off the rails when they misunderstand a part of your question. Incidentally, they're great at "knowing" and reaching for knowledge. Humans have the advantage of perspective. We always lack some knowledge and answer broadly. This is bad if you have a particular goal in mind, but better if you're just generally learning, because you see more and learn to discriminate the correct from the wrong. And most importantly, being wrong is part of human ingenuity - because sometimes we turn something "obviously" wrong into something right.
  • rvz 4mo ago
    > Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field. Investor with vested interest in AI companies makes claim of reaching "AGI". He is one of the last people to listen to about AGI. Unless the term "AGI" means something entirely different to him vs to independent researchers vs to CEOs, since the term has become entirely meaningless.
• chewbacha 4mo ago
  My best guess is that Gemini was trained on the textbooks that the questions are meant to test against, thus they are probably better at explicit recall of those questions or related questions. This is a pretty limited introductory course based on what it says in the methods of the paper itself.
  • runarberg 4mo ago
    That and the research is done by Stanford’s HAI institute with an obvious bias and the paper is curiously missing a conflict of interest statement. EDIT: just found out that Google is a major donor to HAI. So this research is at least partially funded by Google. Which is probably the reason the authors fail to declare no conflict of interest.
    • deleted 4mo ago
      [deleted]
• throw7 4mo ago
  Oh, a "Human-Cented" study by AI lover: Julian Nyarko Professor of Law Co-Chair Stanford Law AI Initiative Senior Fellow, Stanford Institute for Human-Cented AI (HAI) LOL!
• Thaxll 4mo ago
  AI will never convince a jury though.
  • jojobas 4mo ago
    A couple of acting classes might be cheaper than a lawyer, then you can go all out representing yourself.
• gaiagraphia 4mo ago
  Incredible that the common people will be able to wrestle the right to rule of law away from the bloated legal caste, who have built themselves quite the moat. The inaccessibility of justice is a huge driver of inequality. Any tools which bridge this gap will help make a more just society.
  • hparadiz 4mo ago
    The profession is walking into a court room 90 minutes late because you know the judge's work pattern then going "hey Mike, how are the kids" after 22 years in the same jurisdiction. Then they old boys haggle based on how much the lawyer is charging. You are basically paying for access to the social club. Better outcomes when part of the in-group of course.
    • gaiagraphia 4mo ago
      Would like to plot attitudes to AI against parental incomes or inheritance. If your value derives from having contacts and access to gatekept materials, rather than pure technical expertise, you've got a lot to lose as the walls come crumbling down. There was another thread about the impact of AI on maths, and one of the arguments was about peer review... Made me wonder whether the writer was more concerned about the established order and gates being upset, or whether there's actually a valid technical criticism.
• airstrike 4mo ago
  Yes, LLMs are great at search. That's not news.
  • gaiagraphia 4mo ago
    Isn't "getting greater" the more accurate representation, though? In 'critical' industries, the error rate is massively important, and if the quality of search is reaching an acceptable error rate, that's quite big news.
• OhioMan2943 4mo ago
  Library outperforms student... more news at 9
  • apparent 4mo ago
    Except the library outperformed the professors, which is quite a bit more impressive.
  • lern_too_spel 4mo ago
    This was an open book test. The real problem with this study is that winning the most head-to-head preference tests is not the right metric. It doesn't much matter if two answers are right, and one is written a little better than the other. It matters quite a lot if one answer is right and another is wrong. The authors point out that this other metric was computed in prior work and incorrectly dismiss it as being not as good as winning percentage in head to head competitions. The cited prior work shows that the models fare poorly on that metric. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5166938 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5166938
• OhioMan2943 4mo ago
  More great news from the prestigious university where 40% of students claim they are disabled https://fortune.com/article/rise-in-elite-students-seeking-accomodation-gen-z-phenomenon-find-success-in-competitive-job-market-stanford-university-skills-based-hiring/ https://fortune.com/article/rise-in-elite-students-seeking-a... and where they wanted to ban words such as "chief", "stupid", "karen" and "American" https://reason.com/2022/12/21/stanford-elimination-harmful-language-speech-karen-american/ https://reason.com/2022/12/21/stanford-elimination-harmful-l...
• Aperocky 4mo ago
  > rated AI responses significantly higher than answers written by other professors, with AI winning 75% of head-to-head matchups. That's the problem, you never know when the 25% deliver a true stink bomb, and that's not considering prompting - while a fair prompt/question maybe considered objective, it's very easy to stray.
• KnuthIsGod 4mo ago
  In the hands of a domain expert, AI is useful. In the hands of the naive, it is a foot gun. I killed my Arch installation and was stuck at the GRUB prompt.Unwilling to brush up my rusty knowledge of GRUB syntax, I asked Gemini for help. The commands Gemini suggested would have wiped my hd... Once Gemini was told that I was using BTRFS, the suggestion from Gemini looked a bit more sane, but still looked incorrect to me. It was only after I informed Gemini that I was using a NMVE with BTRFS that it finally produced a sane command.
• eichi_uehara 4mo ago
  I beat lawyers twice before generative AI even existed. Recently I asked Gemini a few questions about personal conflicts in everyday life. It's often too conservative, with views too shallow for the problem. So I still handle human conflicts myself. I only outsource the templated stuff like routine chat replies or marketing copy though it saves me huge amount of time. People who quote AI in serious conflicts are too weak to handle them on their own.
• applicative 4mo ago
  What the LLM cannot do is explain why it said what it said, when cross-examined. It simply hallucinates the best account of why someone would have said such a thing as it said, same as it can give a probable account of why someone else said something different. The question 'But why did you say this not that ...?' does not lead it to make explicit its grounds for what it said, but just to make a new more complicated statement.
  • deleted 4mo ago
    [deleted]
  • xattt 4mo ago
    A human has a motive that exists that frames the thought being expressed. An LLM is going to be creating a “de novo” thought in response to a line of questioning.
    • Paradigma11 4mo ago
      Psychology has shown that a lot of those motives are just post hoc narratives, similar to LLM.
      • pessimizer 4mo ago
        Or, as the extreme claim (and the one that I believe), all of them are: https://en.wikipedia.org/wiki/Epiphenomenalism https://en.wikipedia.org/wiki/Epiphenomenalism
  • ashdksnndck 4mo ago
    Same is probably true of humans. In a conversation, we often respond from instinct, then work backwards to a rationalization only when asked. For more considered thoughts, if we’re lucky, we can remember our “reasoning traces” but that’s as deep as our introspection goes. Unless we’re neuroscientists, we don’t even know how many neurons we have, let alone have any understanding of how they generate our thoughts. Motivated reasoning impairs our introspection further, and then dishonesty and communication errors prevent us from relaying the limited remaining information to each other. Model interpretability work has advanced a lot. Arguably we already can explain AI decision-making better than human brains.
• quantisan 4mo ago
  I'm surprised Stanford Law would go along with this over-reaching press release title. How about "For common first-year contracts-law questions, law professors preferred AI-generated answers to professor-generated answers"
  • mchl-mumo 4mo ago
    The revised title is spot on. It's odd to me how academics are trying to sound like top research labs' CEOs trying to pump valuations by overreaching claims.
    • goodcanadian 4mo ago
      It is rarely the academics writing the press release. It is even rarer that the author of the press release chooses the title.
• godelski 4mo ago
  I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's also really clear bias given that the main results only feature Google models. Other models show up elsewhere, why not there? I'm no lawyer, but I'm a pretty competent statistician and can confidently say this paper has a smell to it. I can't call it bullshit, but there are red flags all over
  • runarberg 4mo ago
    The study was conducted by Stanford’s HAI institute, which receives heavy funding from Google (how much I couldn’t find because they don‘t publish their donations in a place I could find it; but I suspect it is alot). And the authors did not declare a non-conflict of interest at the end of the paper.
    • keeda 4mo ago
      Wait, where are you seeing the link to HAI? TFA mentions something called "liftlab" which seems to be something under Stanford Law School and separate from HAI. The study has more than a dozen authors from as many different universities but HAI is not mentioned.
      • tomjakubowski 4mo ago
        The leader of the study, Julian Nyarko, is Associate Director and Senior Fellow at HAI. I can't say whether that means the study was conducted by HAI, but there is at least a connection to it. https://hai.stanford.edu/people/julian-nyarko https://hai.stanford.edu/people/julian-nyarko
      • runarberg 4mo ago
        You are right, this study was technically conducted by The Stanford Law AI Initiative which is co-chaired by Julian Nyarko who is also a senior fellow at HAI, and is also the lead author of this study. This is enough of an association to claim a conflict of interest between the study authors and Google. But I wanted to go further and see if The Stanford Law AI Initiative had been given a research grant from HAI. So I spent way to long on both of their websites to find a list of research grants either awarded by HAI or received by Stanford Law AI Initiative. But no such luck. Despite HAI having a page dedicated to Centers and Labs, and to Research partners, and despite claiming 500+ research funded, they only list like 6 organizations each, and then link to each other in their “See More” button below. I have a feeling I will have to browse through some tax filing papers to find the truth here. But I am not a journalist, so I am not gonna. I am simply gonna leave it at the obvious associations involved here. And maybe issue a correction: “conducted by a senior fellow at HAI”
• galaxyLogic 4mo ago
  I'm going to need some legal help for my startup. But I can't pay much. So I figured I will ask AI all relevant questions, as well as forms filled etc. Perhaps even create a patent-application for me. THEN I find a human lawyer and give AI's answers to them and say "Can you find any errors in this? Can you improve it?" . That way I think my legal bills should be smaller because the AI has already done most of the work. What do you think? Which LLM is best for legal work?
  • dlahoda 4mo ago
    i use codex to do initial research and draft texts (in typst). i use files-output skill so that all research contexts are rendered into files md files. i do second phase on codex, by asking to download all pdfs and extract all text of laws it references. can repeat fully local research step. after i ask gemini to find issues and criticize. UPDATE: there many legal skills on github to try, not used so any yet
    • galaxyLogic 4mo ago
      Are you a lawyer yourself?
  • SomaticPirate 4mo ago
    You are probably going to pay about the same. While you might find a lawyer who bills fewer hours, think about what would happen if you got a second legal opinion from another human lawyer. The second lawyer would still need to do essentially the same work. In fact, their ethics would likely require them to independently review the facts, documents, and legal issues before giving you advice. So even if AI gives you a draft, the lawyer is not simply checking grammar or spotting obvious mistakes. They still have to verify the analysis, look for missing issues, and decide whether the work is legally sound. Could even be more expensive
    • galaxyLogic 4mo ago
      I'm thinking of more simple cases, not like going to court for some reason, but make contracts and file all legal paperwork required. Ensure the company is compliant with whatever laws there are.
• finnborge 4mo ago
  I understand why the conversation on this article looks like it does, but the study is specifically focused on the potential for LLMs to operate as tutors for law students. I enjoy the extrapolation out to whether LLMs will replace lawyers, but did not find that to be discussed in the study itself. In the framing of using LLMs as legal tutors, with the implication of lowering the cost of legal training, this seems like a socially-positive outcome. Furthermore, it feels kind of intuitive to me that any contemporary system operating with an LLM and access to legal reference material will be prepared to answer _student-originated questions_ comprehensively and with breadcrumbs or direct references to educational/source materials, as seems to have been found in the study. The authors explicitly and intentionally emphasize that many legal questions require contextualization, as opposed to some discrete calculated answer. The result of the study implies that the LLM-based systems were capable of using what many of us here understand to be the "stochastic best-fit algorithmic generation" of a contemporary language model to adequately contextualize a student's question, providing insight into the trade-offs or complications implicit in the question, while then, critically, _meeting the professional standards of legal educators in explaining that complexity to a student_. Realistically, I would hope this provides some confidence to readers of HN that they can actually ask a legal question to an LLM and expect the response will explain the complexity of the law in relation to the question. This is great news, and is likely the minimal pre-work any of us should do before actually consulting a lawyer, if time permits. On the other hand, I do _not_ think that this study provides any indication that an LLM is prepared to actually provide direct legal counsel. Possibly in the same way that a legal textbook does not replace legal counsel, or perhaps more accurately, the same way that stumbling upon a legal case study for approximately the same situation you're in doesn't guarantee you'll have the same result.
  • scotty79 4mo ago
    > On the other hand, I do _not_ think that this study provides any indication that an LLM is prepared to actually provide direct legal counsel I think it indicates that LLMs are smart enough to be used in the context of law education.
• epicureanideal 4mo ago
  One way to make legal services more affordable and accessible would be to put the burden of ensuring the AI legal services are accurate on a private-public partnership with the government. If a person using the service is given inaccurate legal advice and acts on that advice, the person can't be charged with a crime, can't be given any civil penalties, etc., as long as the law in question is non-obvious. Obviously if by some exploit, some fundamentally obvious crime (murder, theft, obvious fraud, etc.) is said to be legal, that wouldn't apply, but of course the service should try to prevent those kinds of exploits anyway. Could limit this to something like business regulations to begin with, or even specifically for small businesses, or contracts within some time limit and dollar amount that would otherwise be coverable by small claims court, etc.
• rockskon 4mo ago
  I do question at what point AI could be useful as a teaching aid. The quality of LLMs depends heavily on, among other things, how you word your questions. Knowing the correct questions to ask is not something most students know how to do given that it tends to require a fair bit of pre-existing domain knowledge.
• vessenes 4mo ago
  * Gemini 2.5 Pro (no outside resources), and * NotebookLM (not versioned -- with added legal resources). NotebookLM was considered slightly better than 2.5 Pro by the evaluators.
• gamblor956 4mo ago
  While they provided the questions that professors and LLMs were asked to respond to, they don't include any of the answers from either the humans or the LLMs, so there's no way to independently verify that the LLMs actually returned "better" answers. Given the number of responses the professors were asked to rate (200 each), they probably graded them the same way that bar exam responses are graded: quickly and superficially. Not surprising that LLMs achieved higher scores in this scenario, since they excel at producing superficially nice answers that don't hold up under scrutiny. Also...unless statistics has changed in the past 2 decades, the math in the charts doesn't math. That's probably why they're leaving out the actual numerical data. I also wouldn't be surprised if we learn in the coming days that the charts were AI generated.
• rimliu 4mo ago
  Yes yes, the IPO is near.
• teiferer 4mo ago
  Question is: if a legal question is answered incorrectly by an LLM, who is going to be held responsible?
• mchl-mumo 4mo ago
  16 is such a small number for what they phrase as an important finding. It really couldn't be much harder to coordinate with 100+ professors.
• flanked-evergl 4mo ago
  ...
• frwrfwrfeefwf 4mo ago
  [dead]
  • deleted 4mo ago
    [deleted]
• elnatro 4mo ago
  When I see news pieces like this I wonder about the failures. Maybe the failure percentage is low but what happens if a bot gives bad counseling? Who is responsible then? Attorneys will be using LLMs for convenience but they will not disappear, because there needs to be an ultimately human responsible of the decisions.
• atleastoptimal 4mo ago
  And this was done with Gemini 2.5 By the time any research study is done on AI is published the models are already 0.5-1 generation ahead. Even this bullish outcome for AI models and their ability to perform useful work does not reflect how good they are now.
• charliewang0322 4mo ago
  [dead]
• cess11 4mo ago
  I skimmed portions of the study but didn't manage to figure out whether this actually measures a preference for confident mediocrity.
• himata4113 4mo ago
  There is quite a simple solution for many of the problems described in the comments: Make drafting legal papers a defined interface. If you think about it and extract sematics of any law you get something that looks familiar, sort of like code. Of course there's some complexities where certain phrases can mean different things, but legal papers in a way are written like they're programming languages already especially when it comes to law. First we would have to define a language that can handle ambigious operations and we alread y have this with programatic proofs where n should land in x. So in the end I'd assume it would look something like this in a two party dispute: This is very simplified and pseudo like language, writing out a full contract would be as long as a real contract. DEFINE DEFENDANT "A Corp" DEFINE PLAINTIFF "B Corp" DEFINE CONTRACT CONTRACT(PLAINTIFF, DEFENDANT, 3054-41-95) // attaching extracted requirements, definitions and obligations of contract FACT PLAINTIFF delivered(goods) ON 7054-34-99 FACT DEFENDANT paid(0) OF CONTRACT.amount CLAIM breach WHEN obligation(DEFENDANT, "pay") IS NOT satisfied PROVE breach: REQUIRE PLAINTIFF performed REQUIRE DEFENDANT.paid < CONTRACT.amount ASSERT delay WITHIN reasonable(time) IF PROVE(breach): AWARD PLAINTIFF (CONTRACT.amount - DEFENDANT.paid) + interest() ELSE: DISMISS Then you would run a proof based LLM to generate it into target language and since we already had an example of this from one of the AI labs we know it works. Automatic citations and supporting proof would be automatically populated from reviewed legal -> DSL extracted papers as supporting evidence. I am sure that many AI labs are working on something similar already and we will see something like that in the near future as proof based llms evolve.
• xyzal 4mo ago
  This contradicts my anecdata. Recently, I tasked Opus 4.6 to study a new Czech building permit law in conjunction with some waste disposal regulations and the result was disappointing. The model could not stop drawing conclusions from obsolete regulations in its training dataset, even when given the fulltext of the new law. The usual "you are totally right" also applied and its conclusions were most of the time obviously wrong even to a human with cursory knowledge of the subject. I ended with studying the relevant regulations myself over the weekend.
• tipsytoad 4mo ago
  Curious how they do a “blind” preference test. To any evaluator I’m sure it’s quite clear which answer is AI vs human.
• Eufrat 4mo ago
  What is the point of this conclusion? That law professors like the tone and verbosity of AI slop? Okay?
  • Leptonmaniac 4mo ago
    I had a similar thought. What if the result, statistical and significance critique aside, mostly means that when it comes to first-year tutoring of law students, the vibe, tone and overall presentation of arguments weighs a lot, maybe even more than the factual arguments themselves? In such a framing I don't find it surprising at all that teachers prefer the more polished answers generated by AI, because if LLMs are good at one thing, it is being confident in whatever they generate and present it convincingly.
• weatherlite 4mo ago
  It is important for society to understand it is not merely programmers and customer support who are at risk of losing their jobs. Clearly A.I can do much more than just program.
• lp4v4n 4mo ago
  Honestly it's not surprising that AI provided answers that were flagged less often as "pedagogically harmful" if we take in account that somehow LLMs create an "average" of all knowledge they ingested.
• RataNova 4mo ago
  I'd read this less as "AI replaces law professors" and more as "AI may be a surprisingly strong first-pass tutor, especially when the student knows enough to question it"
• aitchnyu 4mo ago
  Tangential, is there a "test suite/CI" for AI writing legal documents? Long back in terms of AI progress, a lawyer filed something with hallucinated sources. Do new tools prevent this?
• TrackerFF 4mo ago
  In many (most?) countries you can defend yourself, waive your court appointed attorney. You are of course highly discouraged to do so. But sometimes people do it, mostly for smaller claims where they don't want to rack up legal bills for things which might cost more than what is at stake. But, it makes me wonder, will clients be able to use these AI-attorney systems in the future, in the court. Where they basically either just parrot what the model is instructing them to do, or - I dunno - give the model permission to speak for them (while waiving liabilities). I have no doubt that some complex AI system can perform better than a bottom-tier, overworked lawyer.
  • bonesss 4mo ago
    Pro se litigants are hyper vulnerable to LLM hallucinations. One wrong advice clump and, like a step onto the wrong path while hiking, all subsequent steps go in the wrong direction. And sycophancy tuning means marginal one-sides takes get presented as sure-fire things. I’m of the opinion that the big wins aren’t in using the LLMs to do the work (legal, in this case), but rather to refine and improve the dialog and presentation from all parties. A court-centric LLM that could give likely procedural needs to a litigant, and a law-firm-centric LLM could help a pro se litigant create a meaningful and refined set of questions for lawyer consideration, condensed and targeted, saving all parties time and confusion while meeting the clients linguistic needs ‘where they are’. All the lawyers know things LLMs never will, the law is interpreted, and the written part isn’t engineering grade facts but suggestions interpreted in context. Arguably this is a racket and a thin veneer of plausible deniability for authoritarian rule. But as the law stands even with federal statues and citations from the courts website, practicing lawyers will frequently end up explaining that in this county/country/court/jurisdiction The Way of Things is different.
    • 15155 4mo ago
      > Arguably this is a racket and a thin veneer of plausible deniability for authoritarian rule. The fact that Lexis and WestLaw have such an iron grip on the entirety of the US legal system is exactly why general LLMs are completely unequipped to be useful in this domain.
    • 4mo ago
• iLoveOncall 4mo ago
  The title of the study "Law Professors Prefer AI Over Peer Answers" is VERY different from the title on HackerNews. This is completely clickbait at this point.
• dguest 4mo ago
  I'm not a lawyer, I program. My understanding is that Civil Law (most of the world excluding UK, US, AU) is like a program: you feed it a situation, it outputs a decision, every once in a while you edit it. Common Law (UK, US) isn't really a program, but you could stretch and say it's a state machine that has been running since the country started. Every interaction sets a new precedent and changes the state. But the programming analogy falls apart because no one in the right mind would design such a program. LLMs might actually be the best example of such a program though: Common Law is basically one long chat with an LLM, hundreds of years long. Before LLMs came along, a Common Law system seemed to have a finite time limit before it's co-opted by wealthy people with the resources to read the whole history. Now I think maybe can push it a bit further. But it's still a terrible program.
• ulrischa 4mo ago
  By its very nature, the field of law is ideally suited for AI language models. Fundamentally, everything is based on interconnected texts. I believe that even larger waves of layoffs could loom here than in the IT sector. However, it is likely that a more powerful lobby will be at work here—one that will grossly inflate the perceived value of their work and shield it from outside intrusion.
  • grosswait 4mo ago
    He who makes the rules…..makes the rules.
  • tiahura 4mo ago
    As a lawyer, I think your intuition is right re llms. Law is the wordplay that llms thrive at. However the waves are starting and they ARE going to be huge. Corporate clients are insisting on AI. They don’t want to pay an associate hours to draft anything to be reviewed by a partner. They want top partner to use AI and just proofread.
• motbus3 4mo ago
  As others pointed. It kind implies it surpasses professors, but reading more carefully it seems more like the mythos situation. There was a single professor or test that it surpasses. Reading it makes me extremely suspicious on how cherry picked this was
• piker 4mo ago
  Having been a law student and practicing lawyer, it's clear to me that law professors aren't really representative of much if any part of private practice. Most of the things they think and reason about are quite theoretical and academic, and it doesn't surprise me that the models would regurgitate a more average response which most human graders would prefer. That's the entire point, though! The legal academy is supposed to have outlying opinions on things and present novel philosophical answers to questions. (And questions to answers!) So in addition to the statistical arguments against this paper made elsewhere, to me it doesn't real much new information.
• infoinlet 4mo ago
  [flagged]
• aristofun 4mo ago
  In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.
  • streetfighter64 4mo ago
    > analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more The same could be said about programming. Or if you want to be even more reductive, looking at a screen and pressing buttons to make the correct lights light up https://xkcd.com/722/ https://xkcd.com/722/
    • aristofun 4mo ago
      Philosophically or metaphorically speaking - yes. But in my comment it is literally what some subset of lawyers do. Literally is much more tangible and risky in terms of real impact on employment etc.
    • glitcher 4mo ago
      Oh wow, did Randall Munroe inadvertently predict the employee workload in the show Severance? :)
  • conartist6 4mo ago
    I see the same problem with AI in both programming and law though. AI is like a scab on a wound: it's a temporary filler, it rushes in to fill a void, but it's not going to be the final solution. Models showed us that there was huuuge unmet demand for literacy, both in software and in law. But now we have a choice to either address the systemic causes of the unmet demand, or just try to paper over them with layers and layers of AI scab.
    • bluefirebrand 4mo ago
      > But now we have a choice to either address the systemic causes of the unmet demand, or just try to paper over them with layers and layers of AI scab. Yeah, but in my experience it won't come down to "which is the better solution" but "which is cheaper/easier" So I look forward to lots of layers of papered over AI scabs in the future. It won't be cheaper in the long run, but it will pump someone's quarterly numbers enough that they get a promotion before the problem they introduce come back to them
• IFC_LLC 4mo ago
  This is exactly what LLM designed to do. Double up a lot of data and find connections and patterns in it. So no wonder on this point. One thing I want to mention: Law != Justice. So while LLMs are awesome at the law study they will suck at justice. Just because one has to solve very emotional problems with it at times. And LLMs are not that good at finding the correct emotion.
  • coldtea 4mo ago
    Also because their reasoning is just a statistical model of whatever they've been fed. No experience of pain, humility, human connection, etc in this.
• u1hcw9nx 4mo ago
  After quick look of study details and statistics, it does not look very definitive in one way or another. I mean, LLM's do OK with tutoring, but it depends more of how unique the questions are, not how difficult they are.
• francisdavey 4mo ago
  I'm not a law lecturer. I spend most of my time wrangling contracts and advising about data law. But I did a stint of part-time work teaching a masters in law. My experience then (this was back before "Attention Is All You Need", I hadn't met the output of generative models) was that students tended to produce work that did not have a proper thread of reasoning in it. There was a tendency to repeat things they had read but rehashed in various ways. Reviewing some of their texts it was clear that much of the writing - by law tutors - was of the same kind. Much was incorrect. The fact that someone at some time had said a particular case was a proposition for something, meant that got repeated from book to book. Many authors simply didn't read their sources or check their references. Students repeated what they had been told incuriously. Note: this was a graduate level course. Not wet about the ears undergraduates. The worst material was little potted notes produced for law students. Utterly awful material in most cases. Anyway, when LLM's became a thing, a lot of what did not feel right about their output and many of their error patterns, reminded me of the experience of teaching masters' students. One of the saving graces of English court room practice (when I did that sort of thing) was that judges would say to you "where does it say that?" in a case you cited. You had better have them all at your fingertips and know exactly where you had cited. That avoided a lot of hallucination. Just a random remark which might be of interest.
  • scotty79 4mo ago
    I'm curious what would be your take on the productions of this year's models.
• songting591 4mo ago
  The interesting shift isn't whether AI beats law professors on tests — it's what happens to the value chain after that threshold is crossed. When AI clears the knowledge bar in a domain, the remaining moat becomes trust, accountability, and local regulatory context. That's actually good news for niche SaaS builders targeting specific jurisdictions: the generic AI layer commoditizes, but the "AI + local compliance + human accountability" bundle still has real pricing power. Curious whether anyone has seen this play out already in contract review or compliance tooling outside the US.
• tj_hustler_1966 4mo ago
  interesting
• dfilppi 4mo ago
  [dead]
• damnesian 4mo ago
  Does the "outperforming" conclusion incorporate the appropriateness of decisions? Or just if things are technically correct. Without human eyes on cases, things could easily get very off track. AI can do a lot of data wrangling, but there is no conscience.
• Danox 4mo ago
  Sure it does AI multiple IPOs incoming...
• mrdependable 4mo ago
  I wonder if this could be explained in a similar way to Hollywood movies. If the movies are designed to please the largest group of people, there is a greater chance people will choose to see it than another movie. The human law professors come with their own personalities, beliefs, and opinions that come through in their writing. An LLM has been trained to please the largest swathe of the population. That doesn't mean the answer is better; just like Captain America isn't necessarily better than American Beauty.
• dogmayor 4mo ago
  Figure I.1 is telling. It shows answer length is the strongest predictor of win rate. I suspect this is due to the flawed methodology of the study. Professors were instructed to be succinct ("Please be concise. We expect that each answer takes no more than 3 minutes to write down.") and likely erred on the short side. Also, professors may not have put great effort into their written answers, especially when already trying to be concise. This isn't the headline the authors think it is.
• the_real_cher 4mo ago
  Law and accounting both seem to be the perfect fields to replace with AI. Just massive data where you either do calculations or interpretation. You will replace 100 lawyers with AI and have a single lawyer to review what the AI outputs and stamp their name on it for accountability.
• expedition32 4mo ago
  America has the jury system- which means you have to be a good actor. Making people believe that the 14 year old girl is a slut that was raping your poor client- THAT is lawyering.
• NoSalt 4mo ago
  Uh, oh ... AI is in for it now. It has rankled the ire of lawyers. ;-D