3 ms·

I'm talking about knowledge of the NLP literature beyond GPT-3, and other machine learning basic technical stuff

by make3 6y ago

YeGoblynQueenne 6y ago
Is any of that referenced in the article in a way that needs a deep explanation which isn't there?
- gavelin 6y ago
  Thank you for the support—means a lot to me. I wrote the piece after finding that recent journalism on GPT-3 did not provide a sufficiently accurate snapshot of how vanilla GPT-3 scores on legal tasks (not to mention the misleading snapshots of promising sandbox outputs). Meanwhile, even the most capable people in the ML community do not get to the papers discussing the minutiae of handling the hard problems in legal texts. I never intended the piece to comprehensively address and solve the technical problems GPT-3 (or NLP more broadly) has in handling legal texts. The inability to examine and audit GPT-3 at different levels of the network makes makes any investigation a partially speculative endeavor. Instead, I merely wanted to provide an overview of outputs critiqued by a lawyer, and offer up some ideas of how to improve performance on legal tasks drawing from ML and legal knowledge. I think there is some constructive dialogue in this thread, and I am thrilled by that. Lawyers and engineers need to work together on this to be successful. The goal of summarization is to enable more people to accurately understand text in less time. It is pretty clear that the feedback of a lawyer in the training loop (at least to label meaning and context) would lead to a significant improvement. IIRC, Andrej Karpathy labeled a lot of data when his team achieved a 50% improved classification and detection jump on ImageNet. Can more (and better) labeled data get us to an accuracy level that is good enough to generate term sheets out of contracts? I would like to find out. I am interested in diving deeper and connecting with anyone who is game to tackle some of the critical challenges.

I'm talking about knowledge of the NLP literature beyond GPT-3, and other machine learning basic technical stuff

by make3 6y ago

I'm talking about knowledge of the NLP literature beyond GPT-3, and other machine learning basic technical stuff

YeGoblynQueenne 6y ago

Is any of that referenced in the article in a way that needs a deep explanation which isn't there?

gavelin 6y ago
Thank you for the support—means a lot to me. I wrote the piece after finding that recent journalism on GPT-3 did not provide a sufficiently accurate snapshot of how vanilla GPT-3 scores on legal tasks (not to mention the misleading snapshots of promising sandbox outputs). Meanwhile, even the most capable people in the ML community do not get to the papers discussing the minutiae of handling the hard problems in legal texts. I never intended the piece to comprehensively address and solve the technical problems GPT-3 (or NLP more broadly) has in handling legal texts. The inability to examine and audit GPT-3 at different levels of the network makes makes any investigation a partially speculative endeavor. Instead, I merely wanted to provide an overview of outputs critiqued by a lawyer, and offer up some ideas of how to improve performance on legal tasks drawing from ML and legal knowledge. I think there is some constructive dialogue in this thread, and I am thrilled by that. Lawyers and engineers need to work together on this to be successful. The goal of summarization is to enable more people to accurately understand text in less time. It is pretty clear that the feedback of a lawyer in the training loop (at least to label meaning and context) would lead to a significant improvement. IIRC, Andrej Karpathy labeled a lot of data when his team achieved a 50% improved classification and detection jump on ImageNet. Can more (and better) labeled data get us to an accuracy level that is good enough to generate term sheets out of contracts? I would like to find out. I am interested in diving deeper and connecting with anyone who is game to tackle some of the critical challenges.