4 ms·
> f"Extract the company name from: {text}" I think one thing that's lost in all of the LLM tooling is that it's LLM-or-nothing and people have lost knowledge o
by nkozyra 6mo ago
> f"Extract the company name from: {text}"
I think one thing that's lost in all of the LLM tooling is that it's LLM-or-nothing and people have lost knowledge of other ML approaches that actually work just fine, like entity recognition.
I understand it's easier to just throw every problem at an LLM but there are things where off-the-shelf ML/NLP products work just as well without the latency or expense.
- sbpayne 6mo agoOh 100%! There are many problems (including this one!) that probably aren't best suited for an LLM. I was just trying to pick a really simple example that most people would follow.
- rao-v 6mo agoIs there a non-tranformer based entity extraction solution that's not brittle? My understanding is that the cutting edge in entity extraction (e.g. spaCy) is just small BERT models, which rock for certain things, but don't have the world knowledge to handle typos / misspellings etc.
- roadside_picnic 6mo ago> like entity recognition As someone who has done traditional NLP work as at least part of my job for the last 15 years, LLMs do ofter a vastly superior NER solution over any previous NLP options. I agree with your overall statement, that frequently people rush to grab an LLM when superior options already exist (classification is a big example, especially when the power of embeddings can be leveraged), but NER is absolutely a case where LLMs are the superior option (unless you have latency/cost requirements to force you to choose and inferior quality as the trade off, but your default should be an LLM today).
- mark_l_watson 6mo agoI agree! I used 'symbolic AI' for NLP starting in the early 1980s. Everything back then was so brittle, and very labor intensive.
- Centigonal 6mo agoAlso agree. I spent so much time messing with fuzzy matching libraries and NERs for various entity resolution tasks, collecting and cleaning lists of various entity types, and so forth. IMO you really need a model with the encoded world knowledge of an LLM to reliably and flexibly make determinations like that "WMT" and "wally world" are referring to the same corporate entity.
- Legend2440 6mo agoI don't think you realize how bad NLP was prior to transformers. Oldschool entity recognition was extremely brittle to the point that it basically didn't work. CV too for that matter, object recognition before deep learning required a white background and consistent angles. Remember this XKCD from only 2014? https://xkcd.com/1425/ https://xkcd.com/1425/
- nkozyra 6mo agoCV is a space where I would 100% agree with you. But - edge cases notwithstanding - there's not so much of a dropoff with NER that I would first go to an LLM.
- swyx 6mo agobut then u run into edge cases with indirect references and entity recognition models arent smart enough to deal with them, and bitter lesson hits you again.
- sbpayne 6mo agothe bitter lesson comes for us all, unfortunately!