14 ms·

I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 ex

I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make them much, much more productive. A lot of office work is clicking through sheets and doing manual labeling. It's truly wild what is becoming accessible! Really excited to see the next gen software that the open community comes up with :)

tyingq 3y ago
Agree, though plain old Bayesian classifiers have been able to handle some significant portion of that office work for a long time. And not much ever came from it for everyday stuff outside of spam filters. Maybe both the buzz factor and broader applicability means it's more likely to happen this go around?
- yacine_ 3y ago
  More like: the accessibility is what will make it go around. The accessibility is what changes everything! Getting an easy to use interface (instructions) over python changes the accessibility from the denizens of this website to ~anyone with a computer If you're interested, see this paper that argues that point: https://arxiv.org/abs/2302.06541 https://arxiv.org/abs/2302.06541 Essentially, being label efficient is more important than being compute efficient, because the biggest computing constraint we have is enough humans doing the labeling (and knowing how to work a jupyter notebook), not tensor smashing nvidia cards
  - lisasays 3y ago
    Re: https://arxiv.org/abs/2302.06541 https://arxiv.org/abs/2302.06541 - how are we supposed to take the phrase "agile text classification" seriously? They could have just said "efficient". But no - they had to go for "agile".
- londons_explore 3y ago
  I think the real benefits will be for those with an ad-hoc task and no programming/scripting ability. Sure, you and I know how to write a little script to sort a directory of documents into "schoolwork" and "other stuff". But most people don't have that ability, so giving them that would really help accessibility.
- travisjungroth 3y ago
  Ease of use is huge, so is deployment. Even as a software engineer the overhead for a random classification is so much lower.
- xbmcuser 3y ago
  Ease of use is what is going to change everything. Using natural language to ask something and getting an answer is different from what we had before. I knew I could automate a lot of my paperwork with scripts but as I am not a programer I never gave it more than a cursory look. Last Dec while playing around with ChatGPT I was able to get it to write some python scripts that resulted in my spending less than 20-25 min on tasks that I was spending 20-25 hours on. Now could I have written the scripts myself probably but it would have taken me months whereas with chat gpt it took me 2-3 hours to get a working scripts and another 1-2 hours to optimise them.
rcme 3y ago
LLMs as general purpose classifiers is a really big deal, especially because you can give them fuzzy instructions. I know people are worried about LLMs and spam, but I think LLMs may provide an opportunity to elevate online discourse by being more efficient at filtering out spam and low quality commentary.
- gamegoblin 3y ago
  I already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site. Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.
  - 4dayworkweek4u 3y ago
    [flagged]
  - darkgreene 3y ago
    Do you have a repo you can share? I'd be very interested in running my own local copy
    - tough 3y ago
      +1
  - artificial 3y ago
    This sounds great, would love to see some more about this endeavor.
  - metalliqaz 3y ago
    How do you define the cost function for 'low effort'?
    - gamegoblin 3y ago
czbond 3y ago
Can you explain what you did set up wise for your test? I'm following this "space" but the exact, simple pipelines are eluding me.
- yacine_ 3y ago
  fsreadfilesync and json
ticviking 3y ago
The big thing for me and many others is the ability to use the tool without sending NDA data to a 3rd party. The potential amplifying power of that is enormous.
syntaxing 3y ago
Super curious how you did this! Doesn’t 30B model require a hefty computer to run locally (assuming you’re tuning a non-quantized version)
- drdaeman 3y ago
  Not at all. Even a Raspberry Pi would do - you only need ~6GiB RAM for a 4-bit quantized LLaMA model (though it's gonna be quite slow). A decent modern desktop machine would do just fine, no need for anything extra fancy. What I'm wondering is how they fed the documents, as all those LLMs have limitations on the input sizes.
  - yyyk 3y ago
    >~6GiB RAM That's for the 7B model. The 30B model needs 24GB quantized (or 64GB for the unquantized model).
  - jstarfish 3y ago
    I've seen reports that it wrecks RPi SD cards in short order though, so beware... > What I'm wondering is how they fed the documents, as all those LLMs have limitations on the input sizes. It's like file hashing at scale, you don't have to read the whole stream for every file, just the first 1024/2048 bytes (or first few paragraphs). (This works for classification and sorting, less so for summarization.)
- yacine_ 3y ago
  I'm running 30b quant on my 3090. that many 4bits fortunately fit into my precious vram
bob1029 3y ago
> Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Binary classification can actually take you all the way in terms of classification if you are clever with set theory. It's also one of the most traceable & deterministic ways to understand how the natural language is being interpreted at each step. The amount of performance required to run something like an SVM is laughable compared to what is required to run even baby-tier LLMs. If you can reduce the cost of running models to a <1ms invocation over a few megabytes of black box, you can easily test thousands of these per-user-query. Re-training and iterating is much more enjoyable for these reasons. You also don't need any GPUs for this. At the end of the day, the quality of your data will be the biggest issue with older techniques. LLMs can bandaid all sorts of weird things that crop up in the real world and aren't present in the training data. SVMs cannot tolerate requests delivered in the format of Shakespeare (if unexpected). In a well-controlled domain, you would probably be able to get away with much cheaper options that are also more flexible.
hospitalJail 3y ago
What model are you using? What program are you using? Curious how you run the model then interface with it.
alden5 3y ago
What makes it so much better than normal text classification for me is it doesn't require tons of training data to accurately classify text. using it to parse craigslist posts which i might find interesting showed very promising results although it's fairly slow on my base m1 machine.

tyingq 3y ago

Agree, though plain old Bayesian classifiers have been able to handle some significant portion of that office work for a long time. And not much ever came from it for everyday stuff outside of spam filters. Maybe both the buzz factor and broader applicability means it's more likely to happen this go around?

yacine_ 3y ago
More like: the accessibility is what will make it go around. The accessibility is what changes everything! Getting an easy to use interface (instructions) over python changes the accessibility from the denizens of this website to ~anyone with a computer If you're interested, see this paper that argues that point: https://arxiv.org/abs/2302.06541 https://arxiv.org/abs/2302.06541 Essentially, being label efficient is more important than being compute efficient, because the biggest computing constraint we have is enough humans doing the labeling (and knowing how to work a jupyter notebook), not tensor smashing nvidia cards
- lisasays 3y ago
  Re: https://arxiv.org/abs/2302.06541 https://arxiv.org/abs/2302.06541 - how are we supposed to take the phrase "agile text classification" seriously? They could have just said "efficient". But no - they had to go for "agile".
londons_explore 3y ago
I think the real benefits will be for those with an ad-hoc task and no programming/scripting ability. Sure, you and I know how to write a little script to sort a directory of documents into "schoolwork" and "other stuff". But most people don't have that ability, so giving them that would really help accessibility.
travisjungroth 3y ago
Ease of use is huge, so is deployment. Even as a software engineer the overhead for a random classification is so much lower.
xbmcuser 3y ago
Ease of use is what is going to change everything. Using natural language to ask something and getting an answer is different from what we had before. I knew I could automate a lot of my paperwork with scripts but as I am not a programer I never gave it more than a cursory look. Last Dec while playing around with ChatGPT I was able to get it to write some python scripts that resulted in my spending less than 20-25 min on tasks that I was spending 20-25 hours on. Now could I have written the scripts myself probably but it would have taken me months whereas with chat gpt it took me 2-3 hours to get a working scripts and another 1-2 hours to optimise them.

rcme 3y ago

LLMs as general purpose classifiers is a really big deal, especially because you can give them fuzzy instructions. I know people are worried about LLMs and spam, but I think LLMs may provide an opportunity to elevate online discourse by being more efficient at filtering out spam and low quality commentary.

gamegoblin 3y ago
I already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site. Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.
- 4dayworkweek4u 3y ago
  [flagged]
- darkgreene 3y ago
  Do you have a repo you can share? I'd be very interested in running my own local copy
  - tough 3y ago
    +1
- artificial 3y ago
  This sounds great, would love to see some more about this endeavor.
- metalliqaz 3y ago
  How do you define the cost function for 'low effort'?
  - gamegoblin 3y ago

czbond 3y ago

Can you explain what you did set up wise for your test? I'm following this "space" but the exact, simple pipelines are eluding me.

yacine_ 3y ago
fsreadfilesync and json

ticviking 3y ago

The big thing for me and many others is the ability to use the tool without sending NDA data to a 3rd party. The potential amplifying power of that is enormous.

syntaxing 3y ago

Super curious how you did this! Doesn’t 30B model require a hefty computer to run locally (assuming you’re tuning a non-quantized version)

drdaeman 3y ago
Not at all. Even a Raspberry Pi would do - you only need ~6GiB RAM for a 4-bit quantized LLaMA model (though it's gonna be quite slow). A decent modern desktop machine would do just fine, no need for anything extra fancy. What I'm wondering is how they fed the documents, as all those LLMs have limitations on the input sizes.
- yyyk 3y ago
  >~6GiB RAM That's for the 7B model. The 30B model needs 24GB quantized (or 64GB for the unquantized model).
- jstarfish 3y ago
  I've seen reports that it wrecks RPi SD cards in short order though, so beware... > What I'm wondering is how they fed the documents, as all those LLMs have limitations on the input sizes. It's like file hashing at scale, you don't have to read the whole stream for every file, just the first 1024/2048 bytes (or first few paragraphs). (This works for classification and sorting, less so for summarization.)
yacine_ 3y ago
I'm running 30b quant on my 3090. that many 4bits fortunately fit into my precious vram

bob1029 3y ago

> Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Binary classification can actually take you all the way in terms of classification if you are clever with set theory. It's also one of the most traceable & deterministic ways to understand how the natural language is being interpreted at each step. The amount of performance required to run something like an SVM is laughable compared to what is required to run even baby-tier LLMs. If you can reduce the cost of running models to a <1ms invocation over a few megabytes of black box, you can easily test thousands of these per-user-query. Re-training and iterating is much more enjoyable for these reasons. You also don't need any GPUs for this. At the end of the day, the quality of your data will be the biggest issue with older techniques. LLMs can bandaid all sorts of weird things that crop up in the real world and aren't present in the training data. SVMs cannot tolerate requests delivered in the format of Shakespeare (if unexpected). In a well-controlled domain, you would probably be able to get away with much cheaper options that are also more flexible.

hospitalJail 3y ago

What model are you using? What program are you using? Curious how you run the model then interface with it.

alden5 3y ago

What makes it so much better than normal text classification for me is it doesn't require tons of training data to accurately classify text. using it to parse craigslist posts which i might find interesting showed very promising results although it's fairly slow on my base m1 machine.