3 ms·

> I think there is a good argument in saying simple NLP is basically solved In my experience LLMs can get about 70-80% accuracy on a bunch of NER and text clas

> I think there is a good argument in saying simple NLP is basically solved In my experience LLMs can get about 70-80% accuracy on a bunch of NER and text classification tasks if you give it a reasonable prompt. That's not nothing and it's something that you can get started with super quickly. But you'll have slow responses and typically a 3rd party running the inference. Annotating data yourself to about 2000-3000 examples, on the datasets that I ran my benchmarks on, may get you closer to 80-90%. You'll typically also get fast inference that you can run on your own hardware no problem. By annotating the data myself I also like to think that I understand the problem much better as a consequence. Don't get me wrong. LLMs are cool and interesting ... but they don't seem to replace old-school methods just yet.

mxkopy 3y ago
I doubt LLMs will ever replace old-school methods precisely for this reason: > By annotating the data myself I also like to think that I understand the problem much better as a consequence. I’m sure that once we peer into the black box, we’ll find just more refined old-school methods.
2sk21 3y ago
Indeed, one of my last big projects I worked on before I retired was using an LSTM model for named entity recognition. It was many times faster than anything you can do with an LLM.
pletnes 3y ago
Good points. This means you can use LLMs to create an 80% accurate dataset, then manually correct the last 20% to get all the way.
- john-radio 3y ago
  "Manually" anything just depends on the size of your dataset.
- jgalt212 3y ago
  Manually correcting the wrong 20% may be a reasonable amount of work, but you must examine all 100 to find the 20 that needs fixing. And that is most likely not a reasonable amount of work.
  - pletnes 3y ago
    I’ve done image annotation for ML / computer vision. I found it extremely useful to use my first annotations to train a poor model, then use results from that to annotate new data. You get feedback on model quality as you go, and looking at 100 images with multiple objects is way less work than annotating 100 images. If it’s worth it? I guess that depends on the project you’re working on.

mxkopy 3y ago

I doubt LLMs will ever replace old-school methods precisely for this reason: > By annotating the data myself I also like to think that I understand the problem much better as a consequence. I’m sure that once we peer into the black box, we’ll find just more refined old-school methods.

2sk21 3y ago

Indeed, one of my last big projects I worked on before I retired was using an LSTM model for named entity recognition. It was many times faster than anything you can do with an LLM.

pletnes 3y ago

Good points. This means you can use LLMs to create an 80% accurate dataset, then manually correct the last 20% to get all the way.

john-radio 3y ago
"Manually" anything just depends on the size of your dataset.
jgalt212 3y ago
Manually correcting the wrong 20% may be a reasonable amount of work, but you must examine all 100 to find the 20 that needs fixing. And that is most likely not a reasonable amount of work.
- pletnes 3y ago
  I’ve done image annotation for ML / computer vision. I found it extremely useful to use my first annotations to train a poor model, then use results from that to annotate new data. You get feedback on model quality as you go, and looking at 100 images with multiple objects is way less work than annotating 100 images. If it’s worth it? I guess that depends on the project you’re working on.