7 ms·
> This really should not be a surprise, because even the standard-issue ChatGPT can pass the Bar Exam No, it can’t. The two things that together have sometime
by dragonwriter 4y ago
> This really should not be a surprise, because even the standard-issue ChatGPT can pass the Bar Exam
No, it can’t.
The two things that together have sometimes gotten misrepresented that way in “game of telephone” presentations are:
(1) that when tested on the multiple choice component of the multistate bar exam (not the whole bar exam), it got passing grades in two subjects (evidence and torts), not the whole multiple choice section; which is very much not the same thing as being able to pass the exam, and 50.3% overall (better than chance, since its four choices per question, but also very much not passing.) https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4314839 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4314839
(2) that a set of law professors gave it the exams for four individual law courses and it got an average of C+ scores on those exams, which are minimally-passing grades (but not on the bar exam.) https://www.reuters.com/legal/transactional/chatgpt-passes-law-school-exams-despite-mediocre-performance-2023-01-25/ https://www.reuters.com/legal/transactional/chatgpt-passes-l...
- version_five 4y agoI know this isnt the same, but: I've long had it on my list of things to try to train a classifier to predict the answer to multiple choice tests based on embedding of the questions. Many tests I've seen don't require actual intelligence to pass, just a plausible answer relative to the question phrasing
- tshaddox 4y agoA lot of multiple choice tests are only there to provide some minimum bar of plausible deniability for the examiner. The multiple choice section of a driving test (at least in the two U.S. states I've taken them) is a great example. The questions are almost entirely of the form: Which of these activities is legally permitted while operating an automobile? A. Wearing your seatbelt. B. Being intoxicated. C. Driving 120 miles per hour. D. Intentionally colliding with pedestrians. That way, when someone drives drunk, it clearly isn't the fault of the examiner, because they clearly verified that the person knew that was illegal! (Or at least, if they got that question wrong, they got enough other ones correct.)
- bombcar 4y agoSome questions will be gimmies like that but others will be “which is the default speed for a residential street: 15, 25, 35, 45”
- tshaddox 4y agoWhen I took the CA written exam, the only questions I missed were one or two about the specific penalties for certain infractions. Which I’m pretty sure is not something I need to know. Someone will undoubtedly tell me at my sentencing.
- klabb3 4y agoCorrect me if I’m wrong but aren’t virtually all tests and exams designed to minimize ambiguity, make them fair or easy to grade and questions are designed to have a clear correct answer? This is a stark difference to most real-world human activity. Add to the fact that LLMs perform much better on questions with a lot of training data. And also add the hallucinations or more generally: they don’t ask for help or admit they don’t know, they seem unaware of their own confidence levels. No doubt GPT is fascinating and exciting, but boy we're oversubscribing their abilities. LLMs are even worse than crypto (from a fad perspective) because we naturally anthropomorphize their higher level abilities, which are emergent and not well understood even by experts. And we’re about to plug them straight into critical business flows? Bring the popcorn!
- whatshisface 4y ago>Add to the fact that LLMs perform much better on questions with a lot of training data. The answer key can get a 100% on the exam.
- aperrien 4y agoWould a human with a photographic memory count as possessing an answer key?
- whatshisface 4y agoYes, also a human with an answer key.
- blagie 4y ago> And we’re about to plug them straight into critical business flows? Anyone who thinks this is a problem has never managed flesh-and-blood employees, and especially not minimum wage ones. LLMs don't need to be perfect. The bar they need to meet for a lot of work is just not very high. We're also about a year or two into LLMs, and their capabilities are still increasing rapidly. We don't know where their ceiling is. They might plateau where they are, improve slowly, improve linearly, follow Moore's Law, or head off for singularity.
- MichaelZuo 4y agoWell the first paper's authors does say: "While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future."
- JohnFen 4y agoWhich is an overt admission that LLMs can't pass the bar exam right now.
- MichaelZuo 4y agoChatGPT 2.0 being able to do so would still be pretty relevant.
- dragonwriter 4y agoAnd if ChatGPT 2.0 does, than the future article that claims that stock ChatGPT 2.0 can will be justified. But the current article that claims that the stock ChatGPT of today can do that will still be wrong.
- qualudeheart 4y agoYou should see the capacities from Chinese LLMs.
- dragonwriter 4y agoYes, so the researchers, based on ChatGPT’s failure, predict that some other LLM in the new future will pass the same subset of the bar exam that ChatGPT failed to pass. Which is nice, but very much not support for the article’s claim that stock ChatGPT can, already, pass the bar exam. There is a pretty big gap between “passing the bar exam” and “giving researchers a feeling of optimism that some other system will pass a particular subset of the bar exam in the ’near future’”. ChatGPT has demonstrated the ability to do the latter, not the former.
- chatmasta 4y agoTwo articles by Gary Kasparov spring to mind, the first [0] written in 1996 after his victory over Deep Blue: > If the computer makes the same move that I would make for completely different reasons, has it made an "intelligent" move? Is the intelligence of an action dependent on who (or what) takes it? > This is a philosophical question I did not have time to answer. ...and the second written in 1997 after his loss to Deep Blue, entitled "IBM Owes Mankind a Rematch" [1]: > I think this moment could mark a revolution in computer science that could earn IBM and the Deep Blue team a Nobel Prize. [0] https://content.time.com/time/subscriber/article/0,33009,984305,00.html https://content.time.com/time/subscriber/article/0,33009,984... [1] https://content.time.com/time/subscriber/article/0,33009,986394,00.html https://content.time.com/time/subscriber/article/0,33009,986...
- smeagull 4y agoI think this is a general principle that has been accessible since AI began. Having poor and extremely basic rules of thumb can be unreasonably effective. They might fall down in complicated circumstances, but having a "driving AI" that is just cruise control, solves so many cases of that problem. Degenerate solutions to problems can often solve many cases.
- gerdesj 4y ago"No, it can’t." Will it turn up in court - in person? I'm fairly sure that the Bar requires a corpus vivens for its representatives. I've probably got the Latin very badly wrong ...
- croddin 4y agoThere was an attempt at that.. https://news.ycombinator.com/item?id=34529340 https://news.ycombinator.com/item?id=34529340
- dragonwriter 4y agoYeah, the Robot Lawyer is having some legal issues. https://twitter.com/DoNotPayNYSCEF https://twitter.com/DoNotPayNYSCEF
- narrator 4y agoThe other thing is that the legal system is not an API you can spam with until you get a proper response. If you make nonsense arguments in court, or use confabulated law as a lawyer, you will get your law license revoked. The legal system's resources are limited and it doesn't have unlimited time to hear a computer spout clever, but wrong or made up nonsense.
- ncphil 4y agoWell, I _might_ be able to teach an AI how to do effective legal research and writing, or even to do pre and post trial discovery, but representing someone at trial? Not without full immunity from malpractice liability. It would be fun to see how an AI handled cross examination of a hostile witness, or dealt with a response to their hearsay objection like, "res gestae!". Still, a sophisticated AI could be a real help in transactional legal matters like contracts, real estate, taxes, wills and trusts where boilerplate abounds. As far as grades go, my average in law school was C+ (I'd never seen a blue book before that first final -- in college it was all research papers), but I was also one of the 48% who passed a certain state's bar exam in 1982. So there is hope for those AIs who dream of a future in the legal profession. Or maybe they'd just wind up as a sysadmin for a Fortune 200 company, and never regret it.
- bombcar 4y agoI have read enough bad lawsuits to believe an AI could be better than most pro se litigants.
- nullc 4y agoUnfortunately, the AI read the pro se litigants too. ;)
- bee_rider 4y agoI guess eventually a locally-installable ChatGPT-alike will become available. Someone will figure out how to fine-tune it on legal jargon, and then try and use it to represent themselves.
- CamperBob2 4y ago"This talking dog is a dumbass. It can only pass HALF the bar exam. I don't see what all the hype is about." Not sure that's the indictment you and other ChatGPT detractors are presenting it as.
- dragonwriter 4y agoThe claim in the article is that stock GPT can pass the bar exam today. It cannot. Period. That claim is simply false. It is also false that it “can pass HALF the bar exam.” It passed 2 of 7 sections of the MBE, which is used as the multiple choice part of bar exams (which typically also include essay and practical portions)
- CamperBob2 4y agoI stand corrected
- staticman2 4y agoIt looks like it took the practice bar exam questions, not the "bar exam" which is kept secret to avoid cheating since questions may be reused from year to year, much like the SATS.
- thomastjeffery 4y agoThat's not even the important point. ChatGPT can't do anything. It exhibits behavior that was already encapsulated into the semantics of language itself. The only behavior ChatGPT has is to generate semantic continuations from its implicit language model. Every other behavior is a feature of language, not of ChatGPT. Even if ChatGPT could exhibit a passing exam, that would be the feature of carefully curated language in its dataset; not a feature of ChatGPT itself.
- l33t233372 4y agoI really don’t understand the distinction you’re getting at between what ChatGPT is and what ChatGPT does.
- pharmakom 4y agoit can output procedures for other systems that do something, e.g. code.
- elcomet 4y agoIf a human passed the bar exam, it is because he trained for it and the knowledge is encoded in his brain. The distinction you're trying to make doesn't exist. ChatGPT is what it can do.
- thomastjeffery 4y agoThe distinction may not be obvious, but it is there. The reason it is not obvious is that nearly everything you have heard about ChatGPT itself is wrong. The first thing people do to explain what ChatGPT is and does is to personify it. From then on, they are talking about ChatGPT personified, and not ChatGPT as it literally exists. The second thing people do is draw conclusions about the nature and behavior of ChatGPT itself from the narrative they are telling about ChatGPT personified. It's a case of mistaken identity. ChatGPT has a "brain", but the context that "brain" interacts with is semantics not symbolics. A human, when answering a question, interprets the symbols present in the language, then considers them logically. Finally, they formulate an answer, and express that answer with more symbols. ChatGPT does none of that. ChatGPT doesn't even know what sentences, punctuation, or even words are. The only subjects ChatGPT has in mind are short groups of characters: the tokens from the lexical analysis step. ChatGPT reads those tokens (groups of characters) in order, and generates an implicit model from them. That model is like a map: each token is a feature in the landscape. When ChatGPT gets a prompt, it tokenizes it, then checks the map for the closest match. Then it starts at that location, and steps forward, writing out what it sees along the way. That's everything that "ChatGPT as it literally exists" can do. So where does all the behavior come from? It's the content in the map. It's in language itself. ChatGPT's behavior is limited to interacting with that map, but the effect of interacting with that map is where we get all the interesting behavior. Language does not simply encode data: it also encodes instructions and logical relationships. By simply walking through text and feeling the semantic landscape, ChatGPT exhibits the behavior that was already encoded into the symbolic meaning of that text. It accomplished this implicitly without ever defining the meaning of any symbol. It doesn't even know what a symbol is in the first place! So when ChatGPT exhibits the behavior of a person writing correct answers to an exam, it is not behaving like a person at all. It's not interpreting the questions or finding the answers. Instead, it is simply filling the hole in the story with the semantic landscape it sees nearby. If the result is to place answer after question, that is because that data is already present in the training text that ChatGPT was modeled around. Because of this distinction, we can have a much better understanding of what ChatGPT is and isn't capable of. Because language itself holds the features of truth and lie, mistake and success, elegance and verbosity, love and hate, logic and fallacy, defined and abstract, ambiguous and unambiguous, etc. all equal, ChatGPT must rely on the implementation of language - what was written in the first place - to exhibit behaviors we want it to exhibit. But there is a critical flaw in that. Language allows, and even depends on, ambiguity. The context that resolves ambiguity can exist in many semantic shapes, so a model cannot be guaranteed to choose the semantic content that contains the disambiguation. We haven't solved the context dependence problem of natural language. We have only moved it. ChatGPT's success is dependent entirely on the content it is given. It cannot change its behavior to improve that system.
- twawaaay 4y agoI think people are missing an important nuance when it comes to this "AI passed X exam" series of stories. Can you say a bookshelf or a search engine passes a bar exam because you can ask it any question and you can find an answer there? Does a natural language interface to said bookshelf/search engine make the difference? Storing and retrieving facts is not enough to be a lawyer. Examination systems are built with an assumption an already intelligent person is taking it and verifying that this already intelligent person also learned knowledge necessary to do their job. So even if AI could technically pass the bar exam it does not mean it is good enough to be a lawyer. It is not a general intelligence that can solve a variety of problems, it was just trained to remember a library of fats that a lawyer may need to know but is not enough to make a lawyer from a non-sentient program.
- sarabande 4y agoI don't know about in the US, but I suspect it is similar to Germany, where the bar exam also requires you to argue/reason about/solve novel cases, i.e. not regurgitation of stored information. The AI likely wouldn't pass that.