4 ms·

You are not pushing it at 100. I can classify "Is 20 bottles of ferric chloride' a service or product in probably 2 seconds with a 4090. Something that most

You are not pushing it at 100. I can classify "Is 20 bottles of ferric chloride' a service or product in probably 2 seconds with a 4090. Something that most people don't realize is you can run multiple inference. So with something like a 4090, some solid few shots, and instead of having it classify one example at a time, you can do 5. We can probably run 100 parallel inference at 5 at a time. For about a rate of 250 a second on a 4090. So in 11 hours I'll be done. I'm going with a 7-8B model too. Some of the 1.5-3B models are great and will even run faster. Take a competent developer who knows python and how to use an OpenAI compatible API, they can put this together in 10-15 minutes, with no data science/scikit learn or other NLP toolchain experience. So for personal, medium or even large workloads, I think it has killed it. It needs to be extremely large. If you are classifying or segmenting comments on a social media platform were you need to deal with billions a day, then LLM would be a very inefficient approach, but for 90+% of use cases. I think it wins. I'm assuming you are going to run it locally because everyone is paranoid about their data. It's even cheaper if you use a cloud API.

mikeocool 2y ago
If you have to classify user input as they’re inputting it to provide a response — so it can’t be batched - 2 seconds could potentially be really slow. Though LLMs sure have made creating training data to train old school models for those cases a lot easier.
- griomnib 2y ago
  Yeah, that’s what I do: use LLM to help make training data for small models. It’s ao much more efficient, fast, and ergo, scalable.
WildGreenLeave 2y ago
Correct me if I'm wrong, but, if you run multiple inferences at the same time on the same GPU you will need load multiple models in the vram and the models will fight for resources right? So running 10 parallel inferences will slow everything down 5 times right? Or am I missing something?
- aeternum 2y ago
  No, the key is to use the full context window so you structure the prompt as something like: For each line below, repeat the line, add a comma then output whether it most closely represents a product or service: 20 bottles of ferric chloride salesforce ...
  - e12e 2y ago
    Appreciate the concrete advice in this response. Thank you.
- Palmik 2y ago
  Inference for single example is memory bound. By doing batch inference, you can interleave computation with memory loads, without losing much speed (up until you cross the compute bound threshold).
- bavell 2y ago
  You will most likely be using the same model so just 1 to load into vram.
axegon_ 2y ago
FFS... "Lots of writers, few readers". Read again and do the math: 2 seconds, multiply that by 10 million records which contain this, as well as "alarm installation in two locations" and a whole bunch of other crap with little to no repetition (<2%) and where does that get you? 2 * 10,000,000 = 20,000,000 SECONDS!!!! A day has 86,400 seconds (24 * 3600 = 86,400). The data pipeline needs to finish in <24 hours. Everyone needs to get this into their heads somehow: LLM's are not a silver bullet. They will not cure cancer anytime soon, nor will they be effective or cheap enough to run at massive scale. And I don't mean cheap as in "oh, just get openai subscription hurr durr". Throwing money mindlessly into something is never an effective way to solve a problem.
- gbnwl 2y ago
  Why are you using 2 seconds? The commenter you are responding to hypothesized being able to do 250/s based on "100 parallel inference at 5 at a time". Not speaking to the validity of that, but find it strange that you ran with the 2 seconds number after seemingly having stopped reading after that line, while yourself lamenting people don't read and telling them to "read again".
  - jazzyjackson 2y ago
    OP said 2 seconds as if that wasn't an eternity...
    - gbnwl 2y ago
      But then they said 250/second when running multiple inference? Again I don't know if their assertions about running multiple inference are correct but why focus on the wrong number instead of addressing the actual claim?
      - Vampiero 2y ago
        250/s is still nothing when compared to an actual NLP pipeline that takes a few ms per it, because you can parallelize that too. I know it's hard to understand, but you can achieve a throughput that is a few orders of magnitude higher.
magic_hamster 2y ago
Yes and no. Having used these tools extensively I think it will be some time before LLMs are truly performant. Even smaller models can't be compared to running optimized code with efficient data structures. And smaller models (in general) do reduce the quality of your results in most cases. Maybe LLMs will kill off NLP and other pursuits pretty soon, but at the moment, each have their tradeoffs.
griomnib 2y ago
Or you can build a DistilBERT model and get your egregiously inefficient 2 seconds down to tens of milliseconds.
mystified5016 2y ago
At 2s per query for 10m entries, that's 251 days to run through the database.
vrighter 2y ago
two seconds is a VERY VERY VERY long time. That is mind-bogglingly, insanely slow.
rldjbpin 2y ago
even more naive way - just club several requests into batch of classification requests into one prompt. in practice, this is not production-ready as the llm output does not always contain results for the same number of input (sometimes more than inputted even!)

mikeocool 2y ago

If you have to classify user input as they’re inputting it to provide a response — so it can’t be batched - 2 seconds could potentially be really slow. Though LLMs sure have made creating training data to train old school models for those cases a lot easier.

griomnib 2y ago
Yeah, that’s what I do: use LLM to help make training data for small models. It’s ao much more efficient, fast, and ergo, scalable.

WildGreenLeave 2y ago

Correct me if I'm wrong, but, if you run multiple inferences at the same time on the same GPU you will need load multiple models in the vram and the models will fight for resources right? So running 10 parallel inferences will slow everything down 5 times right? Or am I missing something?

aeternum 2y ago
No, the key is to use the full context window so you structure the prompt as something like: For each line below, repeat the line, add a comma then output whether it most closely represents a product or service: 20 bottles of ferric chloride salesforce ...
- e12e 2y ago
  Appreciate the concrete advice in this response. Thank you.
Palmik 2y ago
Inference for single example is memory bound. By doing batch inference, you can interleave computation with memory loads, without losing much speed (up until you cross the compute bound threshold).
bavell 2y ago
You will most likely be using the same model so just 1 to load into vram.

axegon_ 2y ago

FFS... "Lots of writers, few readers". Read again and do the math: 2 seconds, multiply that by 10 million records which contain this, as well as "alarm installation in two locations" and a whole bunch of other crap with little to no repetition (<2%) and where does that get you? 2 * 10,000,000 = 20,000,000 SECONDS!!!! A day has 86,400 seconds (24 * 3600 = 86,400). The data pipeline needs to finish in <24 hours. Everyone needs to get this into their heads somehow: LLM's are not a silver bullet. They will not cure cancer anytime soon, nor will they be effective or cheap enough to run at massive scale. And I don't mean cheap as in "oh, just get openai subscription hurr durr". Throwing money mindlessly into something is never an effective way to solve a problem.

gbnwl 2y ago
Why are you using 2 seconds? The commenter you are responding to hypothesized being able to do 250/s based on "100 parallel inference at 5 at a time". Not speaking to the validity of that, but find it strange that you ran with the 2 seconds number after seemingly having stopped reading after that line, while yourself lamenting people don't read and telling them to "read again".
- jazzyjackson 2y ago
  OP said 2 seconds as if that wasn't an eternity...
  - gbnwl 2y ago
    But then they said 250/second when running multiple inference? Again I don't know if their assertions about running multiple inference are correct but why focus on the wrong number instead of addressing the actual claim?
    - Vampiero 2y ago
      250/s is still nothing when compared to an actual NLP pipeline that takes a few ms per it, because you can parallelize that too. I know it's hard to understand, but you can achieve a throughput that is a few orders of magnitude higher.

magic_hamster 2y ago

Yes and no. Having used these tools extensively I think it will be some time before LLMs are truly performant. Even smaller models can't be compared to running optimized code with efficient data structures. And smaller models (in general) do reduce the quality of your results in most cases. Maybe LLMs will kill off NLP and other pursuits pretty soon, but at the moment, each have their tradeoffs.

griomnib 2y ago

Or you can build a DistilBERT model and get your egregiously inefficient 2 seconds down to tens of milliseconds.

mystified5016 2y ago

At 2s per query for 10m entries, that's 251 days to run through the database.

vrighter 2y ago

two seconds is a VERY VERY VERY long time. That is mind-bogglingly, insanely slow.

rldjbpin 2y ago

even more naive way - just club several requests into batch of classification requests into one prompt. in practice, this is not production-ready as the llm output does not always contain results for the same number of input (sometimes more than inputted even!)