4 ms·

Spacy [0] is a state-of-art / easy-to-use NLP library from the pre-LLM era. This post is the Spacy founder's thoughts on how to integrate LLMs with the kind of

Spacy [0] is a state-of-art / easy-to-use NLP library from the pre-LLM era. This post is the Spacy founder's thoughts on how to integrate LLMs with the kind of problems that "traditional" NLP is used for right now. It's an advertisement for Prodigy [1], their paid tool for using LLMs to assist data labeling. That said, I think I largely agree with the premise, and it's worth reading the entire post. The steps described in "LLM pragmatism" are basically what I see my data science friends doing — it's hard to justify the cost (money and latency) in using LLMs directly for all tasks, and even if you want to you'll need a baseline model to compare against, so why not use LLMs for dataset creation or augmentation in order to train a classic supervised model? [0] https://spacy.io/ https://spacy.io/ [1] https://prodi.gy/ https://prodi.gy/

famouswaffles 3y ago
>what I see my data science friends doing — it's hard to justify the cost (money and latency) in using LLMs directly for all tasks, and even if you want to you'll need a baseline model to compare against, so why not use LLMs for dataset creation or augmentation in order to train a classic supervised model? The NLP infrastructure and pipelines we have today aren't there because they are necessarily the best way to handle the tasks you want. They're in place because computers simply could not understand text the way we would like and shortcuts, approximations were necessary. Borrowing from the blog, Since you could not simply ask the computer, "How many paragraphs in this review say something bad about the acting? Which actors do they frequently mention?", separate processes of something like tagging names, linking them to a knowledge base, and paragraph-level actor sentiment etc were needed. The approximations are cool and they do work rather well for some use cases but they fall apart in many others. This is why automated resume filtering, moderation etc is still awful with the old techniques. You simply can't do what is suggested above and get the same utility.
PheonixPharts 3y ago
> why not use LLMs for dataset creation or augmentation in order to train a classic supervised model? Or, as I mentioned in another comment, just use the embeddings directly. This also does a lot to remove the "cost (money and latency)" part of the problem since you can batch queries to be lightening fast and the dollar cost of generating the embeddings is effectively zero (~3000 pages of text per $1) for most traditional NLP tasks that require a vector representation.
- azinman2 3y ago
  Embeddings only go so far. A lot of time meaning (especially implicit or contextual) occurs from reading larger portions / knowing related information that will simply not be captured in the embedding.
  - daveguy 3y ago
    If it's not captured in the embeddings the LLM wouldn't "know" it either, right?
- bugglebeetle 3y ago
  How are you batching queries and search in a way that’s “lightning fast”? I’ve not found this to be the case for most vector DBs, especially the building the index part.

famouswaffles 3y ago

>what I see my data science friends doing — it's hard to justify the cost (money and latency) in using LLMs directly for all tasks, and even if you want to you'll need a baseline model to compare against, so why not use LLMs for dataset creation or augmentation in order to train a classic supervised model? The NLP infrastructure and pipelines we have today aren't there because they are necessarily the best way to handle the tasks you want. They're in place because computers simply could not understand text the way we would like and shortcuts, approximations were necessary. Borrowing from the blog, Since you could not simply ask the computer, "How many paragraphs in this review say something bad about the acting? Which actors do they frequently mention?", separate processes of something like tagging names, linking them to a knowledge base, and paragraph-level actor sentiment etc were needed. The approximations are cool and they do work rather well for some use cases but they fall apart in many others. This is why automated resume filtering, moderation etc is still awful with the old techniques. You simply can't do what is suggested above and get the same utility.

PheonixPharts 3y ago

> why not use LLMs for dataset creation or augmentation in order to train a classic supervised model? Or, as I mentioned in another comment, just use the embeddings directly. This also does a lot to remove the "cost (money and latency)" part of the problem since you can batch queries to be lightening fast and the dollar cost of generating the embeddings is effectively zero (~3000 pages of text per $1) for most traditional NLP tasks that require a vector representation.

azinman2 3y ago
Embeddings only go so far. A lot of time meaning (especially implicit or contextual) occurs from reading larger portions / knowing related information that will simply not be captured in the embedding.
- daveguy 3y ago
  If it's not captured in the embeddings the LLM wouldn't "know" it either, right?
bugglebeetle 3y ago
How are you batching queries and search in a way that’s “lightning fast”? I’ve not found this to be the case for most vector DBs, especially the building the index part.