2 ms·
"tl;dr" may very well be the problem! There is a tendency of many tl;dr summaries on the web to oversimplify and skew concepts. If those are included in GPT-3's
by gavelin 6y ago
"tl;dr" may very well be the problem! There is a tendency of many tl;dr summaries on the web to oversimplify and skew concepts. If those are included in GPT-3's dataset, GPT-3's output will try to match the dataset style (according to the parameters) and likely not meet the legal standard. There is a section following the conclusion in the paper where I touch on other ways GPT-3 might be improved for the legal summarization use case. The only way we get there is by delving into the nuance of WHY GPT-3 is not yet good enough to replace lawyers, and HOW we can improve on it as an architecture.
- dweekly 6y agoAbsolutely, and I am grateful for your detailed analysis here. These kinds of spot checks on subjective quality for usability in a domain are a critical part of scoring "are we there yet?" and providing solid exemplars that are motivating to the researchers. It's not hard to imagine a GPT-5 recap in a few years with very different outcomes. Some of the motivation on my side is judging at which point it becomes appropriate to explore having an algorithm provide a summary of a medical conversation, since my day job at Medcorder is building a system that acts as a patient advocate to help them better understand what their doctors are saying. The universal feature request from day one has been summaries (which are fraught if you get them wrong!), so I'm keen to develop a sense of when we're going to get there. (As for the downvotes on my comment above, I'd bet they are, appropriately ironically, from people who themselves didn't actually read the article, since the comment was made with a wink to the fact that the article explicitly evaluates the appropriateness of "tl;dr" summaries and found them wanting.)