4 ms·

I saw the story from yesterday on GPT taking the MBE and thought it was interesting: https://news.ycombinator.com/item?id=34307843 https://news.ycombinator.com/

I saw the story from yesterday on GPT taking the MBE and thought it was interesting: https://news.ycombinator.com/item?id=34307843 https://news.ycombinator.com/item?id=34307843 However, over half of the bar exam is an essay-form test, which is where GPT seems to have made the most advancement, so I thought it would be interesting to give it an actual essay questions and judge its response.

ilaksh 4y ago
This is what I got: https://aidev.codes/u/ilaksh/law1/edit https://aidev.codes/u/ilaksh/law1/edit I tried to give it some hints to give a better answer. Note that this site uses text-davinci-003 with temperature 0. Similar to ChatGPT but not the same model.
- mminer237 4y ago
  Sadly, while that's a bit more logical, that's even more wrong on the law. Recording deeds isn't ever time-barred and and being a grantor's daughter doesn't satisfy notice. I will admit that's very creative though haha. It's regurgitating less of the question, but pulling in the wrong laws to replace it.
  - ilaksh 4y ago
    It's not something to be sad about. It just means the model isn't trained as a lawyer and/or isn't pulling from the right knowledgebase.
jamesliudotcc 4y ago
I took the bar exam. I passed, but it was Illinois, which has low standards and high passage rates. Take that for what it is. On the essay portion, the scoring is about 1/3 focused on knowing the legal doctrine that is being tested, and 2/3 on the structure, clarity, and style of the response. (State the law, analyze how the law applies to the facts presented, assess the strength of the counterargument, come to the conclusion, go back and state the conclusion at the beginning.). In Illinois, the passing score is about 65%, so in theory you could pass the essay part while knowing no law at all. Of course, you will not do so well on the multiple choice. I believe a skilled test taker with no law knowledge could triangulate answers to get to around ... 33%? Of course, with statistical learning, and enough data, you could train an LLM to pass the bar. Honestly, given how robotic studying for the bar exam was, I am only surprised it took until 2022-3

ilaksh 4y ago

This is what I got: https://aidev.codes/u/ilaksh/law1/edit https://aidev.codes/u/ilaksh/law1/edit I tried to give it some hints to give a better answer. Note that this site uses text-davinci-003 with temperature 0. Similar to ChatGPT but not the same model.

mminer237 4y ago
Sadly, while that's a bit more logical, that's even more wrong on the law. Recording deeds isn't ever time-barred and and being a grantor's daughter doesn't satisfy notice. I will admit that's very creative though haha. It's regurgitating less of the question, but pulling in the wrong laws to replace it.
- ilaksh 4y ago
  It's not something to be sad about. It just means the model isn't trained as a lawyer and/or isn't pulling from the right knowledgebase.

jamesliudotcc 4y ago

I took the bar exam. I passed, but it was Illinois, which has low standards and high passage rates. Take that for what it is. On the essay portion, the scoring is about 1/3 focused on knowing the legal doctrine that is being tested, and 2/3 on the structure, clarity, and style of the response. (State the law, analyze how the law applies to the facts presented, assess the strength of the counterargument, come to the conclusion, go back and state the conclusion at the beginning.). In Illinois, the passing score is about 65%, so in theory you could pass the essay part while knowing no law at all. Of course, you will not do so well on the multiple choice. I believe a skilled test taker with no law knowledge could triangulate answers to get to around ... 33%? Of course, with statistical learning, and enough data, you could train an LLM to pass the bar. Honestly, given how robotic studying for the bar exam was, I am only surprised it took until 2022-3