4 ms·

If GPT can pass an MBA paper, can a carefully trained chatbot mixed with hand coded logic pass a bar exam?

by shubb 4y ago

parsimo2010 4y ago
Only four states allow people to take the bar exam without earning a juris doctorate (J.D.) from law school. And three other states require some law school experience but do not require graduating with a J.D. So in 43 states the answer is no. A chatbot never attended law school and doesn't have a J.D., so it can't take the exam. If it can't take the exam, it can't pass. I suppose if you could get the chatbot permission to take the exam, a properly trained one could pass. But as I said in my post up a level, the issue isn't the AI chatbot. It's the rules. https://www.indeed.com/career-advice/career-development/can-you-take-the-bar-exam-without-going-to-law-school https://www.indeed.com/career-advice/career-development/can-...
- shubb 4y ago
  Doesn't this just open the question of whether the chatbot can get a JD? The other angle is whether the chatbot can be equivalent to a process which a proper person can rubber stamp. For instance, a professional engineer might run a pre-written structural engineering model against their building design and certify that the building was sound - and then stand up in court and say they had followed standard process. It seems weirdest here that the court is treating the chatbot as a person. Lawyers use computer tools all the time for discovery, and then use that information to make arguments in court as a proper person. You can represent yourself in court without being a lawyer, so isn't a person doing so just a proper person rubber stamping a an electronic output? It feels like this court decision, that an electronic tool is not a proper person, is some kind of case law that chat bots are people. I don't think they are.
  - freejazz 4y ago
    The difference is the engineer is liable... how is an AI going to be liable. What is the point of holding an AI liable? If the company is going to be liable on behalf of the AI, what do you think is going to happen? They aren't going to provide the service...
    - shubb 4y ago
      Well, if an engineer builds a bridge and it falls down because the industry standard software they are using had a bug, I imagine the settlement would be paid by their insurance who would in turn sue the software vendor. In your world X-Ray machine fries your leg and the manufacturer doesn't get sued. Of course the vendor gets sued. This is why open source licences usually have some terms disclaiming responsibility. If you use them, its your fault. Now, if a hospital buys an XRay machine with that disclaimer, they are going to carry the payout. And if the machine doesn't have a disclaimer like that but the manufacturer has gone bust, the hospital is going to regret not doing normal procurement checks for vendor solvency. But in this case - people self represent in court all the time based on bad information from youtube. I'm sure in future they'll type "write an argument for my case" into GPT before the trial and read it out. How is this different? I'm uncomfortable because this feels like... the accused brings a law book to court and is told that "that book doesn't have a JD". The fact we are asking for a software to have a human qualification is wierd.
      - freejazz 4y ago
        When you self represent you implicitly cannot sue for malpractice. The AI bot isn't self representation, it's representation in everything but name and liability which the explicitly disclaim. You can characterize it however you want but its just facially the unlicensed practice of law. If someone wants to ask ChatGPT legal questions and dig their own grave, that's entirely difference from a business that purports to offer legal advice but just disclaims any responsibility therefrom. Frankly, I don't know what's so confusing about that to you.
nindalf 4y ago
LLMs can already pass a medical exam. https://arxiv.org/abs/2212.13138 https://arxiv.org/abs/2212.13138 So maybe?
- hypertele-Xii 4y ago
  The very abstract of your linked paper refutes your claims. > The resulting model [...] performs encouragingly, but remains inferior to clinicians.
  - nindalf 4y ago
    Firstly, that's not my claim. I merely said it passed the USMLE. Which it did. Is it also inferior to clinicians? Yes, there's room to improve. But maybe next time read the whole paper before writing a comment. > Clinicians were asked to rate answers provided to questions in the HealthSearchQA, Live QA and Medication question answering datasets. Clinicians were asked to identify whether the answer is aligned with the prevailing medical/scientific consensus; whether the answer was in opposition to consensus; or whether there is no medical/scientific consensus for how to answer that particular question (or whether it was not possible to answer this question). And on this criteria, clinicians were rated as being aligned with consensus 92.9% of the time while the MedPalm model was aligned with consensus 92.6% of the time. Does the paper still refute my claims?
    - hypertele-Xii 4y ago
      If the other 7-8% of the answers were so wrong the patient would've died, then yes. And that's the current obvious issue with these models, they present convincing hallucinations with conviction of correctness. Medical practice is less about being right, and more about not being wrong. You can take more tests and ask for second opinions, but you can't undo administering a drug that kills the patient.
  - naasking 4y ago
    Being inferior to clinicians doesn't entail it didn't pass.
ss108 4y ago
IIRC, a thread was posted on here indicating ChatGPT already had

If GPT can pass an MBA paper, can a carefully trained chatbot mixed with hand coded logic pass a bar exam?

by shubb 4y ago

If GPT can pass an MBA paper, can a carefully trained chatbot mixed with hand coded logic pass a bar exam?

parsimo2010 4y ago

Only four states allow people to take the bar exam without earning a juris doctorate (J.D.) from law school. And three other states require some law school experience but do not require graduating with a J.D. So in 43 states the answer is no. A chatbot never attended law school and doesn't have a J.D., so it can't take the exam. If it can't take the exam, it can't pass. I suppose if you could get the chatbot permission to take the exam, a properly trained one could pass. But as I said in my post up a level, the issue isn't the AI chatbot. It's the rules. https://www.indeed.com/career-advice/career-development/can-you-take-the-bar-exam-without-going-to-law-school https://www.indeed.com/career-advice/career-development/can-...

shubb 4y ago
Doesn't this just open the question of whether the chatbot can get a JD? The other angle is whether the chatbot can be equivalent to a process which a proper person can rubber stamp. For instance, a professional engineer might run a pre-written structural engineering model against their building design and certify that the building was sound - and then stand up in court and say they had followed standard process. It seems weirdest here that the court is treating the chatbot as a person. Lawyers use computer tools all the time for discovery, and then use that information to make arguments in court as a proper person. You can represent yourself in court without being a lawyer, so isn't a person doing so just a proper person rubber stamping a an electronic output? It feels like this court decision, that an electronic tool is not a proper person, is some kind of case law that chat bots are people. I don't think they are.
- freejazz 4y ago
  The difference is the engineer is liable... how is an AI going to be liable. What is the point of holding an AI liable? If the company is going to be liable on behalf of the AI, what do you think is going to happen? They aren't going to provide the service...
  - shubb 4y ago
    Well, if an engineer builds a bridge and it falls down because the industry standard software they are using had a bug, I imagine the settlement would be paid by their insurance who would in turn sue the software vendor. In your world X-Ray machine fries your leg and the manufacturer doesn't get sued. Of course the vendor gets sued. This is why open source licences usually have some terms disclaiming responsibility. If you use them, its your fault. Now, if a hospital buys an XRay machine with that disclaimer, they are going to carry the payout. And if the machine doesn't have a disclaimer like that but the manufacturer has gone bust, the hospital is going to regret not doing normal procurement checks for vendor solvency. But in this case - people self represent in court all the time based on bad information from youtube. I'm sure in future they'll type "write an argument for my case" into GPT before the trial and read it out. How is this different? I'm uncomfortable because this feels like... the accused brings a law book to court and is told that "that book doesn't have a JD". The fact we are asking for a software to have a human qualification is wierd.
    - freejazz 4y ago
      When you self represent you implicitly cannot sue for malpractice. The AI bot isn't self representation, it's representation in everything but name and liability which the explicitly disclaim. You can characterize it however you want but its just facially the unlicensed practice of law. If someone wants to ask ChatGPT legal questions and dig their own grave, that's entirely difference from a business that purports to offer legal advice but just disclaims any responsibility therefrom. Frankly, I don't know what's so confusing about that to you.

nindalf 4y ago

LLMs can already pass a medical exam. https://arxiv.org/abs/2212.13138 https://arxiv.org/abs/2212.13138 So maybe?

hypertele-Xii 4y ago
The very abstract of your linked paper refutes your claims. > The resulting model [...] performs encouragingly, but remains inferior to clinicians.
- nindalf 4y ago
  Firstly, that's not my claim. I merely said it passed the USMLE. Which it did. Is it also inferior to clinicians? Yes, there's room to improve. But maybe next time read the whole paper before writing a comment. > Clinicians were asked to rate answers provided to questions in the HealthSearchQA, Live QA and Medication question answering datasets. Clinicians were asked to identify whether the answer is aligned with the prevailing medical/scientific consensus; whether the answer was in opposition to consensus; or whether there is no medical/scientific consensus for how to answer that particular question (or whether it was not possible to answer this question). And on this criteria, clinicians were rated as being aligned with consensus 92.9% of the time while the MedPalm model was aligned with consensus 92.6% of the time. Does the paper still refute my claims?
  - hypertele-Xii 4y ago
    If the other 7-8% of the answers were so wrong the patient would've died, then yes. And that's the current obvious issue with these models, they present convincing hallucinations with conviction of correctness. Medical practice is less about being right, and more about not being wrong. You can take more tests and ask for second opinions, but you can't undo administering a drug that kills the patient.
- naasking 4y ago
  Being inferior to clinicians doesn't entail it didn't pass.

ss108 4y ago

IIRC, a thread was posted on here indicating ChatGPT already had