5 ms·

Can GPT pass the bar exam?

65 points by mminer237 4y ago

mminer237 4y ago
I saw the story from yesterday on GPT taking the MBE and thought it was interesting: https://news.ycombinator.com/item?id=34307843 https://news.ycombinator.com/item?id=34307843 However, over half of the bar exam is an essay-form test, which is where GPT seems to have made the most advancement, so I thought it would be interesting to give it an actual essay questions and judge its response.
- ilaksh 4y ago
  This is what I got: https://aidev.codes/u/ilaksh/law1/edit https://aidev.codes/u/ilaksh/law1/edit I tried to give it some hints to give a better answer. Note that this site uses text-davinci-003 with temperature 0. Similar to ChatGPT but not the same model.
  - mminer237 4y ago
    Sadly, while that's a bit more logical, that's even more wrong on the law. Recording deeds isn't ever time-barred and and being a grantor's daughter doesn't satisfy notice. I will admit that's very creative though haha. It's regurgitating less of the question, but pulling in the wrong laws to replace it.
    - ilaksh 4y ago
      It's not something to be sad about. It just means the model isn't trained as a lawyer and/or isn't pulling from the right knowledgebase.
- jamesliudotcc 4y ago
  I took the bar exam. I passed, but it was Illinois, which has low standards and high passage rates. Take that for what it is. On the essay portion, the scoring is about 1/3 focused on knowing the legal doctrine that is being tested, and 2/3 on the structure, clarity, and style of the response. (State the law, analyze how the law applies to the facts presented, assess the strength of the counterargument, come to the conclusion, go back and state the conclusion at the beginning.). In Illinois, the passing score is about 65%, so in theory you could pass the essay part while knowing no law at all. Of course, you will not do so well on the multiple choice. I believe a skilled test taker with no law knowledge could triangulate answers to get to around ... 33%? Of course, with statistical learning, and enough data, you could train an LLM to pass the bar. Honestly, given how robotic studying for the bar exam was, I am only surprised it took until 2022-3
152334H 4y ago
Another example of "ChatGPT for open-ended law": https://broodingomnipresence.substack.com/p/brooding-omnipresent-ai https://broodingomnipresence.substack.com/p/brooding-omnipre... Tangentially related: DoNotPay's (probably illegal) plans to use LLMs in actual court sessions https://twitter.com/jbrowder1/status/1612312707398795264 https://twitter.com/jbrowder1/status/1612312707398795264
- wyager 4y ago
  Whether or not it's illegal, the ABA looks out for its own, and they would never permit AI to threaten the comfortable arrangement they've made for members of their industry. The legal profession is not only self-regulating, but since they make up like 95% of legislators, they get to regulate everyone else too! Very masterful, even more so than the AMA
  - bawolff 4y ago
    I highly doubt chatgpt is even remotely near being a lawyer. Lawyering is all about logical argument from a specific context and nitpicky details, which chatgpt is known to be bad at. Anyways it probably should be illegal if someone is selling "legal services" but not doing it properly for the same reason that if a dr just did what chat-gpt said, that should also be illegal.
    - wonderwonder 4y ago
      What if in 10 years that version of it better than a real lawyer? 10 years is a long time in ai today
      - asadlionpk 4y ago
        certainly won't be the transformer models (alone)
      - bawolff 4y ago
        The future is unknowable. I have my doubts, but ultimately that is a bridge to cross when and if we get there. Notwithstanding how impressive chatgpt is, right now we are nowhere near that bridge so no point speculating.
gnicholas 4y ago
I recall from my bar prep class that you get a substantial amount of partial credit if you state a legal test and apply it correctly — even if the test you state is not correct. That is, if you forget one of the prongs of a test, you lose very few points. The same applies if you completely make up a prong and use it in place of one of the actual prongs. If this is still true, then GPT should be able to do well if it is trained to spit out rules (even incorrect ones) and then apply the facts of the scenario to the stated rules.
xwdv 4y ago
We must replace lawyers with AI so we can battle corporations in the courts, cheaply and at scale.
- inetknght 4y ago
  Somehow I don't think that will go as well as you hope it would.
  - eternalban 4y ago
    Does raise the bar. Legislation can set minimum FLOPS for public defenders.
    - tehjoker 4y ago
      I can assure you that the minimum will be set reasonably at first and then be exceeded within a year. Then corporate influence will prevent it from being raised on a reasonable schedule, just like with minimum wage.
      - eternalban 4y ago
        I sometimes omit the /s. I said the state will set the 'minimum' for public defenders, not 'maximum' for 'general AI attorney'.
    - smegger001 4y ago
      I would prefer they legislate it have the same harware specs and access to the same training data as the prosecutor bot.
  - wyldfire 4y ago
    If we follow this incredible path over the coming century, all I can think of is the future doctor from "The Little Black Bag" [1]. Physician's tools become so advanced as to lower the bar for future doctors. Maybe future attorneys could rely more on AI to provide insight into relevant existing and potential novel case law. [1] https://en.wikipedia.org/wiki/The_Little_Black_Bag https://en.wikipedia.org/wiki/The_Little_Black_Bag
zone411 4y ago
I tried this question, but I added "MEE Question 6" at the top so it has a better context and the answer seems somewhat better? No idea if it would get any credit though, maybe the author of this blog post can chime in. > It is likely that Abe is entitled to possession of Whiteacre. The deed that Owen delivered to Abe twenty years ago constituted a valid conveyance of the property to Abe, even though it was never recorded. Abe subsequently took possession of the land and made use of it for the next eleven years, paying the property taxes and allowing others to use the land. This constitutes "adverse possession," which is a legal principle that allows someone to gain ownership of a piece of property through continuous, open, and notorious possession of the property for a certain period of time. > In this case, Abe has satisfied the requirements for adverse possession under the laws of State B, as he has been in possession of the property openly, continuously, and notoriously for more than the statutory period of time required by State B. Doris and Buyer are considered "subsequent purchasers," as they acquired the property after Abe's conveyance to them, and they do not have a valid claim to the property because they did not have notice of Abe's conveyance. Therefore, it is likely that Abe would be entitled to possession of Whiteacre in this case.
braingenious 4y ago
tl;dr: No. > ChatGPT's essay possibly would score zero points. It lacks any correct rules and any logical analysis. I am really getting tired of the “how many angels could fit on the head of a pin?” style navel-gazing articles that get attention strictly because they use the word ChatGPT in the title. Could ChatGPT write some pointless fluff that lands on the front page of HN? Almost certainly.
- totorovirus 4y ago
  I agree. Some losers trying to get cheap attention for writing cheap blog posts
  - ericsoderstrom 4y ago
    twist: these posts are written by chatGPT
- paxys 4y ago
  > Could ChatGPT write some pointless fluff that lands on the front page of HN? Almost certainly. It's sad that despite all the hype I have seen about ChatGPT over the last month it seems that in the real world its only real application is to generate low effort blogspam.
pvg 4y ago
Previous threads from yesterday and a week ago, 150ish comments each: https://news.ycombinator.com/item?id=34307843 https://news.ycombinator.com/item?id=34307843 https://news.ycombinator.com/item?id=34231083 https://news.ycombinator.com/item?id=34231083
seanp2k2 4y ago
Wait until Congress critters get a hold of this and start using it to stuff their bills with seemingly innocuous legal fluff text as a way to bury important policy decisions then force votes before everyone has time to sift through it all.
- redbar0n 4y ago
  I think they already do this.
- deleted 4y ago
  [deleted]
- Gh0stRAT 4y ago
  It works both ways though: >ChatGPT, summarize the following bill and be sure to note any unexpected clauses: <text of bill here>
- Am4TIfIsER0ppos 4y ago
  They don't need to. Do you think any of your politicians read that few thousand page long spending bill before passing it? Maybe you are alleging it already happens. After all the govt has the resources to generate this software before anyone else.
  - smegger001 4y ago
    In some cases I doubt that the politicians can read books without pictures, let alone read multi-thousand page bills written in legal jargon.
rayiner 4y ago
The MBE is a knowledge test. It tests how well you can memorize a bunch of state law that you’ll never use after the bar exam. Constrained by the multiple choice format, I’d suspect an AI could actually do quite well on the MBE portion. But nobody really cares about the bar exam. The exam the industry is actually built around is the LSAT, which decides the rank of law school you get into, which in turn decides what career pathways are available. That’s a logical reasoning test with no knowledge component. It would be interesting to apply GPT to that.
- worldsavior 4y ago
  > It tests how well you can memorize a bunch of state law that you’ll never use after the bar exam. What makes you a good lawyer is if you remember all those laws, and you know how to find them in your mind. Those laws are mostly necessary.
  - tptacek 4y ago
    Lawyers specialize, just like software developers do, and they also look stuff up, just like software developers do.
    - lostlogin 4y ago
      I wonder if lawyers find their subtly different questions marked as duplicates?
    - worldsavior 4y ago
      That doesn't conflict.
      - oneoff786 4y ago
        It does though. You’re not a good programmer for memorizing a bunch of functions. You’re not a good lawyer for memorizing a bunch of laws. Even if they’re the ones you need.
dimatura 4y ago
"it's impressive for what it is, but ..." is probably a good way to describe chatGPT's performance for a lot of tasks - but what's interesting is the vast diversity in these tasks. I've asked it write me a few poems and it does much better than I could. I've asked it to write songs (including chord progressions, instrument/synth suggestions) for specific genres and it was OK, not great. I've seen people use it for fake maury transcripts, essays in the style of X, marketing copy, and as here, answer legal questions (which as far as I can tell it did far better than I could, not having any legal training whatsoever). chatGPT does seem quite a good (though unreliable and perhaps overconfident) generalist. I'm wondering how much better GPTs trained/fine tuned specifically for a given field - say, contract law - would do for that task.

65 points by mminer237 4y ago

I saw the story from yesterday on GPT taking the MBE and thought it was interesting: https://news.ycombinator.com/item?id=34307843 https://news.ycombinator.com/item?id=34307843 However, over half of the bar exam is an essay-form test, which is where GPT seems to have made the most advancement, so I thought it would be interesting to give it an actual essay questions and judge its response.

ilaksh 4y ago
This is what I got: https://aidev.codes/u/ilaksh/law1/edit https://aidev.codes/u/ilaksh/law1/edit I tried to give it some hints to give a better answer. Note that this site uses text-davinci-003 with temperature 0. Similar to ChatGPT but not the same model.
- mminer237 4y ago
  Sadly, while that's a bit more logical, that's even more wrong on the law. Recording deeds isn't ever time-barred and and being a grantor's daughter doesn't satisfy notice. I will admit that's very creative though haha. It's regurgitating less of the question, but pulling in the wrong laws to replace it.
  - ilaksh 4y ago
    It's not something to be sad about. It just means the model isn't trained as a lawyer and/or isn't pulling from the right knowledgebase.
jamesliudotcc 4y ago
I took the bar exam. I passed, but it was Illinois, which has low standards and high passage rates. Take that for what it is. On the essay portion, the scoring is about 1/3 focused on knowing the legal doctrine that is being tested, and 2/3 on the structure, clarity, and style of the response. (State the law, analyze how the law applies to the facts presented, assess the strength of the counterargument, come to the conclusion, go back and state the conclusion at the beginning.). In Illinois, the passing score is about 65%, so in theory you could pass the essay part while knowing no law at all. Of course, you will not do so well on the multiple choice. I believe a skilled test taker with no law knowledge could triangulate answers to get to around ... 33%? Of course, with statistical learning, and enough data, you could train an LLM to pass the bar. Honestly, given how robotic studying for the bar exam was, I am only surprised it took until 2022-3

152334H 4y ago

Another example of "ChatGPT for open-ended law": https://broodingomnipresence.substack.com/p/brooding-omnipresent-ai https://broodingomnipresence.substack.com/p/brooding-omnipre... Tangentially related: DoNotPay's (probably illegal) plans to use LLMs in actual court sessions https://twitter.com/jbrowder1/status/1612312707398795264 https://twitter.com/jbrowder1/status/1612312707398795264

wyager 4y ago
Whether or not it's illegal, the ABA looks out for its own, and they would never permit AI to threaten the comfortable arrangement they've made for members of their industry. The legal profession is not only self-regulating, but since they make up like 95% of legislators, they get to regulate everyone else too! Very masterful, even more so than the AMA
- bawolff 4y ago
  I highly doubt chatgpt is even remotely near being a lawyer. Lawyering is all about logical argument from a specific context and nitpicky details, which chatgpt is known to be bad at. Anyways it probably should be illegal if someone is selling "legal services" but not doing it properly for the same reason that if a dr just did what chat-gpt said, that should also be illegal.
  - wonderwonder 4y ago
    What if in 10 years that version of it better than a real lawyer? 10 years is a long time in ai today
    - asadlionpk 4y ago
      certainly won't be the transformer models (alone)
    - bawolff 4y ago
      The future is unknowable. I have my doubts, but ultimately that is a bridge to cross when and if we get there. Notwithstanding how impressive chatgpt is, right now we are nowhere near that bridge so no point speculating.

gnicholas 4y ago

I recall from my bar prep class that you get a substantial amount of partial credit if you state a legal test and apply it correctly — even if the test you state is not correct. That is, if you forget one of the prongs of a test, you lose very few points. The same applies if you completely make up a prong and use it in place of one of the actual prongs. If this is still true, then GPT should be able to do well if it is trained to spit out rules (even incorrect ones) and then apply the facts of the scenario to the stated rules.

xwdv 4y ago

We must replace lawyers with AI so we can battle corporations in the courts, cheaply and at scale.

inetknght 4y ago
Somehow I don't think that will go as well as you hope it would.
- eternalban 4y ago
  Does raise the bar. Legislation can set minimum FLOPS for public defenders.
  - tehjoker 4y ago
    I can assure you that the minimum will be set reasonably at first and then be exceeded within a year. Then corporate influence will prevent it from being raised on a reasonable schedule, just like with minimum wage.
    - eternalban 4y ago
      I sometimes omit the /s. I said the state will set the 'minimum' for public defenders, not 'maximum' for 'general AI attorney'.
  - smegger001 4y ago
    I would prefer they legislate it have the same harware specs and access to the same training data as the prosecutor bot.
- wyldfire 4y ago
  If we follow this incredible path over the coming century, all I can think of is the future doctor from "The Little Black Bag" [1]. Physician's tools become so advanced as to lower the bar for future doctors. Maybe future attorneys could rely more on AI to provide insight into relevant existing and potential novel case law. [1] https://en.wikipedia.org/wiki/The_Little_Black_Bag https://en.wikipedia.org/wiki/The_Little_Black_Bag

zone411 4y ago

I tried this question, but I added "MEE Question 6" at the top so it has a better context and the answer seems somewhat better? No idea if it would get any credit though, maybe the author of this blog post can chime in. > It is likely that Abe is entitled to possession of Whiteacre. The deed that Owen delivered to Abe twenty years ago constituted a valid conveyance of the property to Abe, even though it was never recorded. Abe subsequently took possession of the land and made use of it for the next eleven years, paying the property taxes and allowing others to use the land. This constitutes "adverse possession," which is a legal principle that allows someone to gain ownership of a piece of property through continuous, open, and notorious possession of the property for a certain period of time. > In this case, Abe has satisfied the requirements for adverse possession under the laws of State B, as he has been in possession of the property openly, continuously, and notoriously for more than the statutory period of time required by State B. Doris and Buyer are considered "subsequent purchasers," as they acquired the property after Abe's conveyance to them, and they do not have a valid claim to the property because they did not have notice of Abe's conveyance. Therefore, it is likely that Abe would be entitled to possession of Whiteacre in this case.

braingenious 4y ago

tl;dr: No. > ChatGPT's essay possibly would score zero points. It lacks any correct rules and any logical analysis. I am really getting tired of the “how many angels could fit on the head of a pin?” style navel-gazing articles that get attention strictly because they use the word ChatGPT in the title. Could ChatGPT write some pointless fluff that lands on the front page of HN? Almost certainly.

totorovirus 4y ago
I agree. Some losers trying to get cheap attention for writing cheap blog posts
- ericsoderstrom 4y ago
  twist: these posts are written by chatGPT
paxys 4y ago
> Could ChatGPT write some pointless fluff that lands on the front page of HN? Almost certainly. It's sad that despite all the hype I have seen about ChatGPT over the last month it seems that in the real world its only real application is to generate low effort blogspam.

pvg 4y ago

Previous threads from yesterday and a week ago, 150ish comments each: https://news.ycombinator.com/item?id=34307843 https://news.ycombinator.com/item?id=34307843 https://news.ycombinator.com/item?id=34231083 https://news.ycombinator.com/item?id=34231083

seanp2k2 4y ago

Wait until Congress critters get a hold of this and start using it to stuff their bills with seemingly innocuous legal fluff text as a way to bury important policy decisions then force votes before everyone has time to sift through it all.

redbar0n 4y ago
I think they already do this.
deleted 4y ago
[deleted]
Gh0stRAT 4y ago
It works both ways though: >ChatGPT, summarize the following bill and be sure to note any unexpected clauses: <text of bill here>
Am4TIfIsER0ppos 4y ago
They don't need to. Do you think any of your politicians read that few thousand page long spending bill before passing it? Maybe you are alleging it already happens. After all the govt has the resources to generate this software before anyone else.
- smegger001 4y ago
  In some cases I doubt that the politicians can read books without pictures, let alone read multi-thousand page bills written in legal jargon.

rayiner 4y ago

The MBE is a knowledge test. It tests how well you can memorize a bunch of state law that you’ll never use after the bar exam. Constrained by the multiple choice format, I’d suspect an AI could actually do quite well on the MBE portion. But nobody really cares about the bar exam. The exam the industry is actually built around is the LSAT, which decides the rank of law school you get into, which in turn decides what career pathways are available. That’s a logical reasoning test with no knowledge component. It would be interesting to apply GPT to that.

worldsavior 4y ago
> It tests how well you can memorize a bunch of state law that you’ll never use after the bar exam. What makes you a good lawyer is if you remember all those laws, and you know how to find them in your mind. Those laws are mostly necessary.
- tptacek 4y ago
  Lawyers specialize, just like software developers do, and they also look stuff up, just like software developers do.
  - lostlogin 4y ago
    I wonder if lawyers find their subtly different questions marked as duplicates?
  - worldsavior 4y ago
    That doesn't conflict.
    - oneoff786 4y ago
      It does though. You’re not a good programmer for memorizing a bunch of functions. You’re not a good lawyer for memorizing a bunch of laws. Even if they’re the ones you need.

dimatura 4y ago

"it's impressive for what it is, but ..." is probably a good way to describe chatGPT's performance for a lot of tasks - but what's interesting is the vast diversity in these tasks. I've asked it write me a few poems and it does much better than I could. I've asked it to write songs (including chord progressions, instrument/synth suggestions) for specific genres and it was OK, not great. I've seen people use it for fake maury transcripts, essays in the style of X, marketing copy, and as here, answer legal questions (which as far as I can tell it did far better than I could, not having any legal training whatsoever). chatGPT does seem quite a good (though unreliable and perhaps overconfident) generalist. I'm wondering how much better GPTs trained/fine tuned specifically for a given field - say, contract law - would do for that task.