10 ms·

Has LLM killed traditional NLP?

154 points by vietthangif 2y ago

deleted 2y ago
[deleted]
RancheroBeans 2y ago
NLP is an important part of upcoming RAG frameworks like Microsoft’s LazyGraphRAG. So I think it’s more like NLP is a tool used when the time is right. https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/ https://www.microsoft.com/en-us/research/blog/lazygraphrag-s...
- politelemon 2y ago
  I could use some help understanding, is this a set of tools or techniques to answer questions? The name made me think it's related to create embeddings but it seems much more?
oliwary 2y ago
This article seems to be paywalled unfortunately. While LLMs are very useful when the tasks are complex and/or there is not a lot of training data, I still think traditional NLP pipelines have a very important role to play, including when: - Depending on the complexity of the task and the required results, SVMs or BERT can be enough in many cases and take much lower resources, especially if there is a lot of training data available. Training these models with LLM outputs could also be an interesting approach to achieve this. - When resources are constrained or latency is important. - In some cases, there may be labeled data in certain classes that have no semantic connection between them, e.g. explaining the class to LLMs could be tricky.
- eminent101 2y ago
  > This article seems to be paywalled unfortunately. I am no fan of Medium paywalled articles but if it helps you, here's the article on archive - https://archive.is/J53CE https://archive.is/J53CE
- 99catmaster 2y ago
  https://archive.is/J53CE https://archive.is/J53CE
michaelsbradley 2y ago
https://archive.is/J53CE https://archive.is/J53CE
axegon_ 2y ago
No, it has not and will not in the foreseeable future. This is one of my responsibilities at work. LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost. LLMs are great at mid-level complexity tasks given a reasonable volume of data - they can take away the tedious job of figuring out what you are looking at or even come up with some basic mapping. But anything at large volumes.. Na. Real life example: "is '20 bottles of ferric chloride' a service or a product?" One prompt? Fair. 10? Still ok. 100? You're pushing it. 10M - get help.
- diggan 2y ago
  So TLDR: You agree with the author, but not for the same reasons?
- vlovich123 2y ago
  That’s the argument the article makes but the reasoning is a little questionable on a few fronts: - It uses f16 for the data format whereas quantization can reduce the memory burden without a meaningful drop in accuracy, especially as compared with traditional NLP techniques. - The quality of LLMs typically outperform OpenCV + NER. - You can choose to replace just part of the pipeline instead of using the LLM for everything (e.g. using text-only 3B or 1B models to replace the NER model while keeping OpenCV) - The (LLM compute / quality) / watt is constantly decreasing. Meaning even if it’s too expensive today, the system you’ve spent time building, tuning and maintaining today is quickly becoming obsolete. - Talking with new grads in NLP programs, all the focus is basically on LLMs. - The capability + quality out of models / size of model keeps increasing. That means your existing RAM & performance budget keeps absorbing problems that seemed previously out of reach Now of course traditional techniques are valuable because they can be an important tool in bringing down costs (fixed function accelerator vs general purpose compute), but it’s going to become more niche and specialized with most tasks transitioning to LLMs I think. The “bitter lesson” paper is really relevant to these kinds of discussions.
  - vlovich123 2y ago
    Not an independent player so obviously important to be critical of papers like this [1], but it’s claiming a ~10x cost in LLM inference every year. This lines up with the technical papers I’m seeing that are continually improving performance + the related HW improvements. That’s obviously not sustainable indefinitely, but these kinds of exponentials are precisely why people often make incorrect conclusions on how long change will take to happen. Just a reminder: CPUs were 2x more performance every 18 months and continued to continually upend software companies for 20 years who weren’t in tune with this cycle (i.e. focusing on performance instead of features). For example, even if you’re spending $10k/month for LLM vs $100/month to process the 10M item, it can still be more beneficial to go the LLM route as you can buy cheaper expertise to put together your LLM pipeline than the NLP route to make up the ~100k/year difference (assuming the performance otherwise works and the improved quality and robustness of the LLM solution isn’t providing extra revenue to offset). [1] https://a16z.com/llmflation-llm-inference-cost/ https://a16z.com/llmflation-llm-inference-cost/
scarface_74 2y ago
For my use case, definitely. I have worked on AWS Connect (online call center) and Amazon Lex (the backing NLP engine) projects. Before LLMs, it was a tedious process of trying to figure out all of the different “utterances” that people could say and the various languages you had to support. With LLMs, it’s just prompting https://chatgpt.com/share/678bab08-f3a0-8010-82e0-32cff9c0b428 https://chatgpt.com/share/678bab08-f3a0-8010-82e0-32cff9c0b4... I used something like this using Amazon Bedrock and a Lambda hook for Amazon Lex. Of course it wasn’t booking a flight. It was another system The above is a simplified version. In the real world , I gave it a list of intents (book flights, reserve a room, rent a car) and properties - “slots” - I needed for each intent.
- gtirloni 2y ago
  How about the the costs?
  - scarface_74 2y ago
    We measure savings in terms of call deflections. Clients we work with say that each time a customer talks to an agent it costs $2-$5. That’s not even taking into account call abandonments
    - IanCal 2y ago
      My base thing while advising people is that if anyone you pay needs to read the output, or you are directly replacing any kind of work then even frontier llm model inference costs are irrelevant. Of course you need to work out of that's truly the case but people worry about the cost in places where it's just irrelevant. If it's $2 when you get to an agent, each case that's avoided there could pay for around a million words read/generated. That's expensive compared to most API calls but irrelevant when counting human costs.
- elicksaur 2y ago
  Thank you for sharing an actual prompt thread. So much of the LLM debate is washed in biases, and it is very helpful to share concrete examples of outputs.
DebtDeflation 2y ago
The question seems malformed to me. Text classification, clustering, named entity recognition, etc. are NLP tasks. LLMs can perform these tasks. ML models that are not LLMs (or even not deep learning models) can also perform these tasks. Is the author perhaps asking if the concept of a "completion" has replaced all of these tasks? When I hear "traditional NLP" I think not of the above types of tasks but rather the methodology employed for performing them. For example, building a pipeline to do stemming/lemmatization, part of speech tagging, coreference resolution, etc. before the text gets fed to a classifier model. This was SOTA 10 years ago but I don't think many people are still doing it today.
- lyu07282 2y ago
  I understand the term the same way, but I don't think "traditional NLP" died with LLMs, traditional NLP already died with deep learning 7ish years ago, with LSTMs and CNNs, the gains from such models trained end-to-end are so huge it just doesn't make sense to chop up the tasks like that.
retinaros 2y ago
cant read the article. do they consider BERT as an LLM? there are tasks still in NLP where BERT is better than a GPT like
- selimthegrim 2y ago
  Like?
  - Olfurm 2y ago
    Like named entity recognition or relations recognition. Check https://github.com/urchade/GLiNER https://github.com/urchade/GLiNER
    - fbilhaut 2y ago
      Indeed. What's is interesting (and not so common) with models like GLiNER is that it is way lighter than LLMs while preserving good (if not better) quality and some zero-shot ability (in this case wrt to the entity classes). This feature is very significant, as most "traditional" (including Transformer-based) approaches are more or less supervised during fine-tuning. Icing on the cake, you don't have to deal with all the problems that arise when you want to get structured output from an LLM (in this case structured NER output).
deleted 2y ago
[deleted]
darepublic 2y ago
I remember using the open NLP library from Stanford around 2016. It would do parts of speech tagging of words in a sentence (labelling the words with their grammatical function). It was pretty good but reliably failed on certain words where context determined the tag. When for gpt 3 came out the first thing I tested it out on was parts of speech tagging. In particular those sentences open NLP had trouble with. And it aced everything I was impressed.
leobg 2y ago
There are AI bros that will call an LLM to do what you could do with a regex. I’ve seen people do the chunking for RAG using an LLM…
- tossandthrow 2y ago
  If you think about chunking as "take x characters" then using LLMs is a poor idea. But syntactic chunking also works really poorly for any serious application as you loose basically all context. Semantic chunking, however, is a task you absolutely would use LLMs for.
  - leobg 2y ago
    If by LLM you mean embeddings I agree. Though you can often get away with using much smaller models for that. I was talking about people who actually make a call to a completion endpoint and then have the LLM repeat the input text token for token just to get the split.
    - tossandthrow 2y ago
      How do you do semantic chunking using embeddings? And yes, I perfectly now what you are talking about. And yes, that is a perfect strategy to chunk large texts so you can index it. It does not sound like you are familiar with chunking and it's current issues?
freefaler 2y ago
If archive links aren't working this works: https://freedium.cfd/https://medium.com/altitudehq/is-traditional-nlp-dead-05544ae7d756 https://freedium.cfd/https://medium.com/altitudehq/is-tradit...
deleted 2y ago
[deleted]
vedant 2y ago
The title of this article feels like "has electricity killed oil lamps"?
- deleted 2y ago
  [deleted]
itissid 2y ago
LLM Design/Use has only about as much to with engineering as building a plane has to do with actually flying it. Every business is kind of a unicorn in its problems NLP is a small part of it. Like even if it did perform cheaply enough to do NLP, how would you replace parts like: 1. Evaluation system that uses Calibration(Human labels) 2. Ground Truth Collection(Human + sometimes semi automated) 3. QA testing by end users. Even if LLMs made it easier to do NLP there are correlations with the above which means your NLP process is hugely influenced so much that you still need an engineer. If you have an engineer who only for doing NLP and nothing else you are quite hyper specialized like to the extent you are only building planes 0.01%: of the engineering work out there.
thangalin 2y ago
I created an NLP library to help curl straight quotes into curly quotes. Last I checked, LLMs struggled to curl the following straight quotation marks: ''E's got a 'ittle box 'n a big 'un,' she said, 'wit' th' 'ittle 'un 'bout 2'×6". An' no, y'ain't cryin' on th' "soap box" to me no mo, y'hear. 'Cause it 'tweren't ever a spec o' fun!' I says to my frien'. The library is integrated into my Markdown editor, KeenWrite (https://keenwrite.com/ https://keenwrite.com/), to correctly curl quotation marks into entities before passing them over to ConTeXt for typesetting. While there are other ways to indicate opening and closing quotation marks, none are as natural to type in plain text as straight quotes. I would not trust an LLM curl quotation marks accurately. For the curious, you can try it at: https://whitemagicsoftware.com/keenquotes/ https://whitemagicsoftware.com/keenquotes/ If you find any edge cases that don't work, do let me know. The library correctly curls my entire novel. There are a few edge cases that are completely ambiguous, however, that require semantic knowledge (part-of-speech tagging), which I haven't added. PoS tagging would be a heavy operation that could prevent real-time quote curling for little practical gain. The lexer, parser, and test cases are all open source. https://gitlab.com/DaveJarvis/KeenQuotes/-/tree/main/src/main/java/com/whitemagicsoftware/keenquotes https://gitlab.com/DaveJarvis/KeenQuotes/-/tree/main/src/mai...
- jcheng 2y ago
  Great example. I just tried it with a few LLMs and got horrible results. GPT-4o got a ton of them wrong, GPT-1o got them all correct AFAICT but took 1m50s to do so, and Claude 3.5 Sonnet said “Here's the text with straight quotes converted to curly quotes” but then returned the text with all the straight quotes intact. I’m very surprised all three models didn’t nail it immediately.
- gf000 2y ago
  I would be interested how well would even a smaller LLM model work after fine tuning. Besides the overhead of an LLM, I would assume they would do a much better job at it in the edge cases (where contextual understanding is required).
derbaum 2y ago
One of the things I'm still struggling with when using LLMs over NLP is classification against a large corpus of data. If I get a new text and I want to find the most similar text out of a million others, semantically speaking, how would I do this with an LLM? Apart from choosing certain pre-defined categories (such as "friendly", "political", ...) and then letting the LLM rate each text on each category, I can't see a simple solution yet except using embeddings (which I think could just be done using BERT and does not count as LLM usage?).
- thaumasiotes 2y ago
  Take two documents. Feed one through an LLM, one word at a time, and keep track of words that experience greatly inflated probabilities of occurrence, compared to baseline English. "For" is probably going to maintain a level of likelihood close to baseline. "Engine" is not. Do the same thing for the other one. See how much overlap you get.
  - derbaum 2y ago
    Wouldn't a simple comparison of the word frequency in my text against a list of usual word frequencies do the trick here without an LLM? Sort of a BM25?
    - thaumasiotes 2y ago
      It might; it's not going to do the same thing. The LLM will tell you words that would likely appear in a similar text. Word frequency will tell you words that have actually appeared in your text. I'm postulating that the first kind of list is much more likely to show strong overlap between two similar documents than the second kind of list. Vocabulary style matters a lot to what words are actually used, but much less to what words are likely to be used. If I'm following a style guide that says to use "automobile" instead of "car", appearance probabilities for "automobile" will be greatly inflated. And appearance probabilities for "car" will also be greatly inflated, just to a lesser extent than for "automobile". Whereas actual usage of "car" will be pegged at zero. Determining how similar two texts are is something that an LLM should be good at. It should be better than a simple comparison of word frequency. Whether it's better enough to justify the extra compute is a different question.
vletal 2y ago
The idea that we can solve "language" by breaking down and understanding sentences is naive and funny with the benefit of hindsight, is it not? An equivalently funny attitude seems to be the "natural language will replace programming languages". Let's see how that one will work out when the hype is over.
- Vampiero 2y ago
  It's not naive, it's how languages work. That grammar doesn't necessarily convey 100% of semantics is a problem of natural language. Or rather, of people being poor at communicating unambiguously. Programming languages can also be ambiguous sometimes, but that ambiguity is resolved before execution by essentially following a priority list or throwing an error if no combination of rules fits.
- Pompidou 2y ago
  They all fall into the pitfall that Martin Heidegger suggested avoiding, particularly in the introduction of his text What Is a Thing?.
ein0p 2y ago
You have to be more specific with that question. The "traditional" NLP has by now been killed twice: first by classical machine learning (which significantly reduced the need for linguists), and now by deep learning (which has all but eliminated it). So "traditional" NLP was killed back in late 00s. You can't kill that which is not alive, so it follows that LLMs have not, in fact, killed traditional NLP.
antonvs 2y ago
One datapoint: we were using NLP to translate natural language instructions into an executable form that could drive our product. It was part of a product we sold to enterprises. We've completely replaced that with LLMs. We still use our own DNNs for certain tasks, but not for NLP.
arcknighttech 2y ago
They also have their own uses depending in the case scenario so it depends on what someone is working with. NLP in some cases like advanced marketing can still have great benefit without an LLM and AIs in general are streamlining certain "speed of information processing" tasks but still struggle with complex "systems thinking". FYI - If anyone doesn't know the difference between the two or has no idea what NLP or an LLM is, this has a good breakdown: https://medium.com/@melindaboone80722/nlp-vs-llm-b339abdc6511 https://medium.com/@melindaboone80722/nlp-vs-llm-b339abdc651...
rolandthomas 2y ago
[dead]

154 points by vietthangif 2y ago

deleted 2y ago

[deleted]

NLP is an important part of upcoming RAG frameworks like Microsoft’s LazyGraphRAG. So I think it’s more like NLP is a tool used when the time is right. https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/ https://www.microsoft.com/en-us/research/blog/lazygraphrag-s...

politelemon 2y ago
I could use some help understanding, is this a set of tools or techniques to answer questions? The name made me think it's related to create embeddings but it seems much more?

oliwary 2y ago

This article seems to be paywalled unfortunately. While LLMs are very useful when the tasks are complex and/or there is not a lot of training data, I still think traditional NLP pipelines have a very important role to play, including when: - Depending on the complexity of the task and the required results, SVMs or BERT can be enough in many cases and take much lower resources, especially if there is a lot of training data available. Training these models with LLM outputs could also be an interesting approach to achieve this. - When resources are constrained or latency is important. - In some cases, there may be labeled data in certain classes that have no semantic connection between them, e.g. explaining the class to LLMs could be tricky.

eminent101 2y ago
> This article seems to be paywalled unfortunately. I am no fan of Medium paywalled articles but if it helps you, here's the article on archive - https://archive.is/J53CE https://archive.is/J53CE
99catmaster 2y ago
https://archive.is/J53CE https://archive.is/J53CE

michaelsbradley 2y ago

https://archive.is/J53CE https://archive.is/J53CE

axegon_ 2y ago

No, it has not and will not in the foreseeable future. This is one of my responsibilities at work. LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost. LLMs are great at mid-level complexity tasks given a reasonable volume of data - they can take away the tedious job of figuring out what you are looking at or even come up with some basic mapping. But anything at large volumes.. Na. Real life example: "is '20 bottles of ferric chloride' a service or a product?" One prompt? Fair. 10? Still ok. 100? You're pushing it. 10M - get help.

diggan 2y ago
So TLDR: You agree with the author, but not for the same reasons?
vlovich123 2y ago
That’s the argument the article makes but the reasoning is a little questionable on a few fronts: - It uses f16 for the data format whereas quantization can reduce the memory burden without a meaningful drop in accuracy, especially as compared with traditional NLP techniques. - The quality of LLMs typically outperform OpenCV + NER. - You can choose to replace just part of the pipeline instead of using the LLM for everything (e.g. using text-only 3B or 1B models to replace the NER model while keeping OpenCV) - The (LLM compute / quality) / watt is constantly decreasing. Meaning even if it’s too expensive today, the system you’ve spent time building, tuning and maintaining today is quickly becoming obsolete. - Talking with new grads in NLP programs, all the focus is basically on LLMs. - The capability + quality out of models / size of model keeps increasing. That means your existing RAM & performance budget keeps absorbing problems that seemed previously out of reach Now of course traditional techniques are valuable because they can be an important tool in bringing down costs (fixed function accelerator vs general purpose compute), but it’s going to become more niche and specialized with most tasks transitioning to LLMs I think. The “bitter lesson” paper is really relevant to these kinds of discussions.
- vlovich123 2y ago
  Not an independent player so obviously important to be critical of papers like this [1], but it’s claiming a ~10x cost in LLM inference every year. This lines up with the technical papers I’m seeing that are continually improving performance + the related HW improvements. That’s obviously not sustainable indefinitely, but these kinds of exponentials are precisely why people often make incorrect conclusions on how long change will take to happen. Just a reminder: CPUs were 2x more performance every 18 months and continued to continually upend software companies for 20 years who weren’t in tune with this cycle (i.e. focusing on performance instead of features). For example, even if you’re spending $10k/month for LLM vs $100/month to process the 10M item, it can still be more beneficial to go the LLM route as you can buy cheaper expertise to put together your LLM pipeline than the NLP route to make up the ~100k/year difference (assuming the performance otherwise works and the improved quality and robustness of the LLM solution isn’t providing extra revenue to offset). [1] https://a16z.com/llmflation-llm-inference-cost/ https://a16z.com/llmflation-llm-inference-cost/

scarface_74 2y ago

For my use case, definitely. I have worked on AWS Connect (online call center) and Amazon Lex (the backing NLP engine) projects. Before LLMs, it was a tedious process of trying to figure out all of the different “utterances” that people could say and the various languages you had to support. With LLMs, it’s just prompting https://chatgpt.com/share/678bab08-f3a0-8010-82e0-32cff9c0b428 https://chatgpt.com/share/678bab08-f3a0-8010-82e0-32cff9c0b4... I used something like this using Amazon Bedrock and a Lambda hook for Amazon Lex. Of course it wasn’t booking a flight. It was another system The above is a simplified version. In the real world , I gave it a list of intents (book flights, reserve a room, rent a car) and properties - “slots” - I needed for each intent.

gtirloni 2y ago
How about the the costs?
- scarface_74 2y ago
  We measure savings in terms of call deflections. Clients we work with say that each time a customer talks to an agent it costs $2-$5. That’s not even taking into account call abandonments
  - IanCal 2y ago
    My base thing while advising people is that if anyone you pay needs to read the output, or you are directly replacing any kind of work then even frontier llm model inference costs are irrelevant. Of course you need to work out of that's truly the case but people worry about the cost in places where it's just irrelevant. If it's $2 when you get to an agent, each case that's avoided there could pay for around a million words read/generated. That's expensive compared to most API calls but irrelevant when counting human costs.
elicksaur 2y ago
Thank you for sharing an actual prompt thread. So much of the LLM debate is washed in biases, and it is very helpful to share concrete examples of outputs.

DebtDeflation 2y ago

The question seems malformed to me. Text classification, clustering, named entity recognition, etc. are NLP tasks. LLMs can perform these tasks. ML models that are not LLMs (or even not deep learning models) can also perform these tasks. Is the author perhaps asking if the concept of a "completion" has replaced all of these tasks? When I hear "traditional NLP" I think not of the above types of tasks but rather the methodology employed for performing them. For example, building a pipeline to do stemming/lemmatization, part of speech tagging, coreference resolution, etc. before the text gets fed to a classifier model. This was SOTA 10 years ago but I don't think many people are still doing it today.

lyu07282 2y ago
I understand the term the same way, but I don't think "traditional NLP" died with LLMs, traditional NLP already died with deep learning 7ish years ago, with LSTMs and CNNs, the gains from such models trained end-to-end are so huge it just doesn't make sense to chop up the tasks like that.

retinaros 2y ago

cant read the article. do they consider BERT as an LLM? there are tasks still in NLP where BERT is better than a GPT like

selimthegrim 2y ago
Like?
- Olfurm 2y ago
  Like named entity recognition or relations recognition. Check https://github.com/urchade/GLiNER https://github.com/urchade/GLiNER
  - fbilhaut 2y ago
    Indeed. What's is interesting (and not so common) with models like GLiNER is that it is way lighter than LLMs while preserving good (if not better) quality and some zero-shot ability (in this case wrt to the entity classes). This feature is very significant, as most "traditional" (including Transformer-based) approaches are more or less supervised during fine-tuning. Icing on the cake, you don't have to deal with all the problems that arise when you want to get structured output from an LLM (in this case structured NER output).

deleted 2y ago

[deleted]

darepublic 2y ago

I remember using the open NLP library from Stanford around 2016. It would do parts of speech tagging of words in a sentence (labelling the words with their grammatical function). It was pretty good but reliably failed on certain words where context determined the tag. When for gpt 3 came out the first thing I tested it out on was parts of speech tagging. In particular those sentences open NLP had trouble with. And it aced everything I was impressed.

leobg 2y ago

There are AI bros that will call an LLM to do what you could do with a regex. I’ve seen people do the chunking for RAG using an LLM…

tossandthrow 2y ago
If you think about chunking as "take x characters" then using LLMs is a poor idea. But syntactic chunking also works really poorly for any serious application as you loose basically all context. Semantic chunking, however, is a task you absolutely would use LLMs for.
- leobg 2y ago
  If by LLM you mean embeddings I agree. Though you can often get away with using much smaller models for that. I was talking about people who actually make a call to a completion endpoint and then have the LLM repeat the input text token for token just to get the split.
  - tossandthrow 2y ago
    How do you do semantic chunking using embeddings? And yes, I perfectly now what you are talking about. And yes, that is a perfect strategy to chunk large texts so you can index it. It does not sound like you are familiar with chunking and it's current issues?

freefaler 2y ago

If archive links aren't working this works: https://freedium.cfd/https://medium.com/altitudehq/is-traditional-nlp-dead-05544ae7d756 https://freedium.cfd/https://medium.com/altitudehq/is-tradit...

deleted 2y ago

[deleted]

vedant 2y ago

The title of this article feels like "has electricity killed oil lamps"?

deleted 2y ago
[deleted]

itissid 2y ago

LLM Design/Use has only about as much to with engineering as building a plane has to do with actually flying it. Every business is kind of a unicorn in its problems NLP is a small part of it. Like even if it did perform cheaply enough to do NLP, how would you replace parts like: 1. Evaluation system that uses Calibration(Human labels) 2. Ground Truth Collection(Human + sometimes semi automated) 3. QA testing by end users. Even if LLMs made it easier to do NLP there are correlations with the above which means your NLP process is hugely influenced so much that you still need an engineer. If you have an engineer who only for doing NLP and nothing else you are quite hyper specialized like to the extent you are only building planes 0.01%: of the engineering work out there.

thangalin 2y ago

I created an NLP library to help curl straight quotes into curly quotes. Last I checked, LLMs struggled to curl the following straight quotation marks: ''E's got a 'ittle box 'n a big 'un,' she said, 'wit' th' 'ittle 'un 'bout 2'×6". An' no, y'ain't cryin' on th' "soap box" to me no mo, y'hear. 'Cause it 'tweren't ever a spec o' fun!' I says to my frien'. The library is integrated into my Markdown editor, KeenWrite (https://keenwrite.com/ https://keenwrite.com/), to correctly curl quotation marks into entities before passing them over to ConTeXt for typesetting. While there are other ways to indicate opening and closing quotation marks, none are as natural to type in plain text as straight quotes. I would not trust an LLM curl quotation marks accurately. For the curious, you can try it at: https://whitemagicsoftware.com/keenquotes/ https://whitemagicsoftware.com/keenquotes/ If you find any edge cases that don't work, do let me know. The library correctly curls my entire novel. There are a few edge cases that are completely ambiguous, however, that require semantic knowledge (part-of-speech tagging), which I haven't added. PoS tagging would be a heavy operation that could prevent real-time quote curling for little practical gain. The lexer, parser, and test cases are all open source. https://gitlab.com/DaveJarvis/KeenQuotes/-/tree/main/src/main/java/com/whitemagicsoftware/keenquotes https://gitlab.com/DaveJarvis/KeenQuotes/-/tree/main/src/mai...

jcheng 2y ago
Great example. I just tried it with a few LLMs and got horrible results. GPT-4o got a ton of them wrong, GPT-1o got them all correct AFAICT but took 1m50s to do so, and Claude 3.5 Sonnet said “Here's the text with straight quotes converted to curly quotes” but then returned the text with all the straight quotes intact. I’m very surprised all three models didn’t nail it immediately.
gf000 2y ago
I would be interested how well would even a smaller LLM model work after fine tuning. Besides the overhead of an LLM, I would assume they would do a much better job at it in the edge cases (where contextual understanding is required).

derbaum 2y ago

One of the things I'm still struggling with when using LLMs over NLP is classification against a large corpus of data. If I get a new text and I want to find the most similar text out of a million others, semantically speaking, how would I do this with an LLM? Apart from choosing certain pre-defined categories (such as "friendly", "political", ...) and then letting the LLM rate each text on each category, I can't see a simple solution yet except using embeddings (which I think could just be done using BERT and does not count as LLM usage?).

thaumasiotes 2y ago
Take two documents. Feed one through an LLM, one word at a time, and keep track of words that experience greatly inflated probabilities of occurrence, compared to baseline English. "For" is probably going to maintain a level of likelihood close to baseline. "Engine" is not. Do the same thing for the other one. See how much overlap you get.
- derbaum 2y ago
  Wouldn't a simple comparison of the word frequency in my text against a list of usual word frequencies do the trick here without an LLM? Sort of a BM25?
  - thaumasiotes 2y ago
    It might; it's not going to do the same thing. The LLM will tell you words that would likely appear in a similar text. Word frequency will tell you words that have actually appeared in your text. I'm postulating that the first kind of list is much more likely to show strong overlap between two similar documents than the second kind of list. Vocabulary style matters a lot to what words are actually used, but much less to what words are likely to be used. If I'm following a style guide that says to use "automobile" instead of "car", appearance probabilities for "automobile" will be greatly inflated. And appearance probabilities for "car" will also be greatly inflated, just to a lesser extent than for "automobile". Whereas actual usage of "car" will be pegged at zero. Determining how similar two texts are is something that an LLM should be good at. It should be better than a simple comparison of word frequency. Whether it's better enough to justify the extra compute is a different question.

vletal 2y ago

The idea that we can solve "language" by breaking down and understanding sentences is naive and funny with the benefit of hindsight, is it not? An equivalently funny attitude seems to be the "natural language will replace programming languages". Let's see how that one will work out when the hype is over.

Vampiero 2y ago
It's not naive, it's how languages work. That grammar doesn't necessarily convey 100% of semantics is a problem of natural language. Or rather, of people being poor at communicating unambiguously. Programming languages can also be ambiguous sometimes, but that ambiguity is resolved before execution by essentially following a priority list or throwing an error if no combination of rules fits.
Pompidou 2y ago
They all fall into the pitfall that Martin Heidegger suggested avoiding, particularly in the introduction of his text What Is a Thing?.

ein0p 2y ago

You have to be more specific with that question. The "traditional" NLP has by now been killed twice: first by classical machine learning (which significantly reduced the need for linguists), and now by deep learning (which has all but eliminated it). So "traditional" NLP was killed back in late 00s. You can't kill that which is not alive, so it follows that LLMs have not, in fact, killed traditional NLP.

antonvs 2y ago

One datapoint: we were using NLP to translate natural language instructions into an executable form that could drive our product. It was part of a product we sold to enterprises. We've completely replaced that with LLMs. We still use our own DNNs for certain tasks, but not for NLP.

arcknighttech 2y ago

They also have their own uses depending in the case scenario so it depends on what someone is working with. NLP in some cases like advanced marketing can still have great benefit without an LLM and AIs in general are streamlining certain "speed of information processing" tasks but still struggle with complex "systems thinking". FYI - If anyone doesn't know the difference between the two or has no idea what NLP or an LLM is, this has a good breakdown: https://medium.com/@melindaboone80722/nlp-vs-llm-b339abdc6511 https://medium.com/@melindaboone80722/nlp-vs-llm-b339abdc651...

rolandthomas 2y ago

[dead]