3 ms·

As a researcher, it's pretty obvious from the language and from the analysis of the author that he's a novice at machine learning who is a lawyer, not a researc

by make3 6y ago

As a researcher, it's pretty obvious from the language and from the analysis of the author that he's a novice at machine learning who is a lawyer, not a researcher in machine learning

gavelin 6y ago
Correct. I hope that meant that this was more accessible than the average write-up (and not less accurate!) :)
YeGoblynQueenne 6y ago
The author seems to have a very clear understanding of how GPT-3 works and their down-to-Earth, plain language analysis is miles away from the wild flights of fancy we're used to reading about GPT-3. As a for instance, they didn't even use the word "understand" once, to refer to what GPT-3 is doing.
- make3 6y ago
  I'm talking about knowledge of the NLP literature beyond GPT-3, and other machine learning basic technical stuff
  - YeGoblynQueenne 6y ago
    Is any of that referenced in the article in a way that needs a deep explanation which isn't there?
    - gavelin 6y ago
      Thank you for the support—means a lot to me. I wrote the piece after finding that recent journalism on GPT-3 did not provide a sufficiently accurate snapshot of how vanilla GPT-3 scores on legal tasks (not to mention the misleading snapshots of promising sandbox outputs). Meanwhile, even the most capable people in the ML community do not get to the papers discussing the minutiae of handling the hard problems in legal texts. I never intended the piece to comprehensively address and solve the technical problems GPT-3 (or NLP more broadly) has in handling legal texts. The inability to examine and audit GPT-3 at different levels of the network makes makes any investigation a partially speculative endeavor. Instead, I merely wanted to provide an overview of outputs critiqued by a lawyer, and offer up some ideas of how to improve performance on legal tasks drawing from ML and legal knowledge. I think there is some constructive dialogue in this thread, and I am thrilled by that. Lawyers and engineers need to work together on this to be successful. The goal of summarization is to enable more people to accurately understand text in less time. It is pretty clear that the feedback of a lawyer in the training loop (at least to label meaning and context) would lead to a significant improvement. IIRC, Andrej Karpathy labeled a lot of data when his team achieved a 50% improved classification and detection jump on ImageNet. Can more (and better) labeled data get us to an accuracy level that is good enough to generate term sheets out of contracts? I would like to find out. I am interested in diving deeper and connecting with anyone who is game to tackle some of the critical challenges.

by make3 6y ago

As a researcher, it's pretty obvious from the language and from the analysis of the author that he's a novice at machine learning who is a lawyer, not a researcher in machine learning

gavelin 6y ago

Correct. I hope that meant that this was more accessible than the average write-up (and not less accurate!) :)

YeGoblynQueenne 6y ago

The author seems to have a very clear understanding of how GPT-3 works and their down-to-Earth, plain language analysis is miles away from the wild flights of fancy we're used to reading about GPT-3. As a for instance, they didn't even use the word "understand" once, to refer to what GPT-3 is doing.

make3 6y ago
I'm talking about knowledge of the NLP literature beyond GPT-3, and other machine learning basic technical stuff
- YeGoblynQueenne 6y ago
  Is any of that referenced in the article in a way that needs a deep explanation which isn't there?
  - gavelin 6y ago
    Thank you for the support—means a lot to me. I wrote the piece after finding that recent journalism on GPT-3 did not provide a sufficiently accurate snapshot of how vanilla GPT-3 scores on legal tasks (not to mention the misleading snapshots of promising sandbox outputs). Meanwhile, even the most capable people in the ML community do not get to the papers discussing the minutiae of handling the hard problems in legal texts. I never intended the piece to comprehensively address and solve the technical problems GPT-3 (or NLP more broadly) has in handling legal texts. The inability to examine and audit GPT-3 at different levels of the network makes makes any investigation a partially speculative endeavor. Instead, I merely wanted to provide an overview of outputs critiqued by a lawyer, and offer up some ideas of how to improve performance on legal tasks drawing from ML and legal knowledge. I think there is some constructive dialogue in this thread, and I am thrilled by that. Lawyers and engineers need to work together on this to be successful. The goal of summarization is to enable more people to accurately understand text in less time. It is pretty clear that the feedback of a lawyer in the training loop (at least to label meaning and context) would lead to a significant improvement. IIRC, Andrej Karpathy labeled a lot of data when his team achieved a 50% improved classification and detection jump on ImageNet. Can more (and better) labeled data get us to an accuracy level that is good enough to generate term sheets out of contracts? I would like to find out. I am interested in diving deeper and connecting with anyone who is game to tackle some of the critical challenges.