5 ms·
LLMs are stunningly good at language tasks: almost all of what us old-timers called NLP is just crushed these days. Summarization, Q&A, sentiment, the list goes
by benreesman 3y ago
LLMs are stunningly good at language tasks: almost all of what us old-timers called NLP is just crushed these days. Summarization, Q&A, sentiment, the list goes on and on. Truly remarkable stuff.
And where there isn’t a bright line around “fact”, and where it doesn’t need to come together like a Pynchon novel, the generative stuff is smoking hot: short-form fiction, opinion pieces, product copy? Massive productivity booster, you can prototype 20 ideas in one minute.
But that’s about where we are: lift natural language into a latent space with some clear notion of separability, do some affine (ish) transformations, lower back down.
Fucking impressive for a computer. But if it can really carry water for an expensive Penn grad?
You’re paying for something other than blindingly insightful product strategy.
- Jeff_Brown 3y agoI wonder how long it takes AI to get good at law. Right now the verbal tasks it excels at are similar to the artistic ones: namely, solving problems with enormous solution spaces that are robust to small perturbations. That is, change a good picture of an angry tree man slightly and it's still probably a good picture of an angry tree man.
- Dr4kn 3y agoDepends what you would classify as good, but IBM Watson is already used in law firms [today](https://www.ibm.com/case-studies/legalmation https://www.ibm.com/case-studies/legalmation) LLMs iare most often best at helping humans do their tasks more effectively, not replacing them completely
- qingcharles 3y agoI've tried using a lot for writing motions. It can actually do a pretty decent job of writing motions, and it can come up with some arguments that you might not have thought of. You just have to ignore all its citations and look everything up yourself, otherwise this: https://www.reuters.com/legal/new-york-lawyers-sanctioned-using-fake-chatgpt-cases-legal-brief-2023-06-22/ https://www.reuters.com/legal/new-york-lawyers-sanctioned-us...
- matheusmoreira 3y agoIsn't ChatGPT getting progressively better scores on medical and law exams? It will probably pass the USMLE and the bar one day. If it doesn't already. It's gonna be interesting.
- Jeff_Brown 3y agoYes and that's amazing -- but law exams resemble programming exams. In the wild, both labors require you to keep a mountain of project-specific context in your head, something that tests like the LSAT cannot evaluate.
- coffeefirst 3y agoYes, but we should expect that, the answers are in its training data. The problem is passing tests are an okay proxy for competence in humans, but if you think of LLMs as a giant library search engine, the thing it is competent at is identifying and regurgitating compiled phrases from its records. Which is awesome. It can't be a doctor.
- benreesman 3y agoRelatedly, a lot of the cap R fanfic crowd points at stuff pretty clearly lifted out of the Metamorphosis of Prime Intellect or whatever. And it’s dope that it can do that! But let’s keep our heads about what it is.
- qingcharles 3y agoTry asking it for specifically NLP-ized text and it does it very well... (but then tells you not to use it as it is "unethical")
- baq 3y agoYou aren't talking about the NLP the GP is talking about.
- benreesman 3y agoOk now I’m /r/TooAfraidToAsk… NLP means something else relevant to probabilistic language models?
- marcosdumay 3y agoNatural language processing. I actually don't know what you thought it meant.
- qingcharles 3y agoYou're right. That'll teach me to reply to comments before I had my morning coffee. For others arriving here: I suspect OP meant Natural Language Processing and I was talking about Neuro-linguistic Programming. I've had my caffeine now.
- aofjfdgnionio 3y agoI don't buy it. LLMs cannot do anything reliably, no matter how constrained the domain. Their outputs are of acceptable quality when back to a person who will use their human brain to paper over the cracks. People can recognize when the output is garbage, figure out minor ambiguities, and subconsciously correct minor factual or logical errors. But I would never feed LLM results directly into another computer program This rules out most traditional NLP tasks.
- benreesman 3y agoI’m sympathetic to the instinct to push back on the absurd boosterism (these things are an existential threat to humanity this year), it’s fucking annoying. But they can do plenty of useful stuff reliably. It’s not “be generally intelligent”, which they are just nothing even remotely close to, but know you don’t dig the LLM hype from that comment? Yeah, they get that every time.
- aofjfdgnionio 3y agoI have tried to use LLMs, namely GPT4 and Llama-2, for sentiment analysis. They did quite poorly. I asked them to identify which sentiments from a list are found in a given text, with output formatted as a comma-separated list. In response I usually got just that. But sometimes I got a prose explanation of the sentiment, a list containing different keywords than requested, a list formatted differently than I wanted, or nothing useful at all. Sentiment analysis is easy. It wasn't the end goal, just a "hello world" example to verify my tools were set up correctly. I ran into unsolvable problems in the tutorial. I have no use for tools which do amazing things sometimes but which cannot be reasoned about and cannot be prevented from producing garbage. Maybe other people will find uses for them, though. I'll keep an open mind and check back in five years.
- FeepingCreature 3y agoWould you feed human output directly into a computer program? I'm just saying, we invented backspace for a reason. LLMs have no backspace. It's insane they work as well as they do.
- 3y ago
- bugglebeetle 3y ago> almost all of what us old-timers called NLP is just crushed these days For this to be true for most production service use cases, LLMs would need to be at least ~10X faster. I generally agree they can be quite good at these tasks, but the performance is not there to do them on large datasets.
- deleted 3y ago[deleted]