6 ms·
LLMs as general purpose classifiers is a really big deal, especially because you can give them fuzzy instructions. I know people are worried about LLMs and spam
by rcme 4y ago
LLMs as general purpose classifiers is a really big deal, especially because you can give them fuzzy instructions. I know people are worried about LLMs and spam, but I think LLMs may provide an opportunity to elevate online discourse by being more efficient at filtering out spam and low quality commentary.
- gamegoblin 4y agoI already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site. Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.
- 4dayworkweek4u 4y ago[flagged]
- darkgreene 4y agoDo you have a repo you can share? I'd be very interested in running my own local copy
- tough 4y ago+1
- artificial 4y agoThis sounds great, would love to see some more about this endeavor.
- metalliqaz 4y agoHow do you define the cost function for 'low effort'?
- gamegoblin 4y agoI give GPT some examples of what I consider low-effort comments and non-low-effort comments, and then I just ask it to do classification. It's quite good at few-shot classification of fuzzy stuff like this.
- jpe90 4y agoThis sounds super cool, I’m very curious how you implemented it. I’m only vaguely familiar with the API.. if I had to guess I would say you send - a system instruction that its job is to filter unwanted content - examples of unwanted content - an instruction like “filter the following html:” For every web request you want to filter, you would re-send all of those messages followed by the page HTML as the final message. Is that close?
- gamegoblin 4y agoClose, but it's a bit more specialized to just work on HN: - Examples of unwanted content - Then I give a large numbered list of comments and ask which numbers should be filtered - The plugin then just deletes those comment nodes from the DOM. If HN ever updates their HTML I will have to tweak this code. The reason to send a large list of comments is just to save on costs. It's cheaper to do it this way than one comment at a time. So the main difference from what you've proposed is GPT never sees the HTML. My code enumerates the comments in the HTML and splices them in to the prompt in a nice numbered list, then does the reverse translation from list number to DOM element in the other direction.
- jpe90 4y agoOh that’s clever! Thanks for answering!
- xk_id 4y agoI don’t know how anyone can trust to run code on their machine that they don’t understand and hasn’t been reviewed by a third party.
- rcme 4y agoIsn't that pretty much all code?
- sliken 4y agoHow many line of codes are in the kernel, drivers, and apps on your laptop/desktop? How sure are you that all that code has been reviewed by a 3rd party? How many CVEs a year impact your laptop/desktop? Do you have any reason to think that increased productivity with LLM assistance will result in lower quality code? Personally I find LLM assistance increases productivity, decreases the penalty of using a more difficult language like rust, and makes it more palatable to spend more (LLM assisted) time writing tests.
- xk_id 4y agoI meant reviewed by a (competent) human; not, audited by a security team/whatever. Generative computer code is in a whole new category of risk, because it can hallucinate a `rm -rf *` that you might overlook if you don't understand the code yourself.
- sliken 4y agoWell at least for now, any non-trivial code is going to be written by a human+AI, likely one function at a time. It's pretty rare for the AI to give me exactly what I want, often I have to tweak my prompts multiple times, and even just give up and write a function based on the useful bits from previous queries. I just don't see it chatbot assisted programming any worse than what we have today.
- gamegoblin 4y agoI understand the code now, I just didn't know how to write a browser plugin and having GPT4 walk me through it was 10x faster than reading a tutorial.
- ChatGTP 4y agoSome negative, even overly negative comments contain good ideas though so maybe it’s not a great idea?
- heyoni 4y agoYour comment has been detected to contain oppositional tones and has thus been filtered out. You are hereby removed from the discourse. /s
- ChatGTP 4y agoLove it!
- 4dayworkweek4u 4y ago[flagged]
- duckmysick 4y agoThat's ok. If it's a good idea, somebody else will talk about in a positive way. Also, unless you are reading every comment in every thread, you are going to miss a few interesting ideas anyway. That's ok too.
- ChatGTP 4y agoStay calm and keep summarising!
- muyuu 4y agospam and especially phishing will become much, much better it will be hard to trust anything at all
- yacine_ 4y agovery simple, just don't read the email
- SV_BubbleTime 4y agoWhich will force drive signed communications. So... win.
- muyuu 4y agoYea in a sense it will be a win, but it will up the stakes. A lot of people with get scammed. A lot more than right now.
- SV_BubbleTime 4y agoDisagree. Adding signed communications as the default to replace email's completely nonsense zero encryption model is not upping the stakes. Email is the most successful federated software in the universe, but it's fucking time to get some encryption and authn.
- muyuu 4y agoyou are having a good laugh if you think the average granny is going to do that anytime soon
- nathan_compton 4y agoLike almost all machine learning stuff, I expect these things to only be useful in places where it doesn't really matter if the results are correct. When you apply a classifier in real life its critically important to understand its statistical characteristics which is typically done via model characterization, which involves cross validation or boot strapping or whatever. I think the idea that you can just zero-shot or few-shot deploy these things as classifiers and forget about it is incredibly naive unless (as I said earlier) the results just don't really matter. I've already used LLMs in my work as a data scientist but it requires a ton of work to just make the results tractable (and I have been using GPT4, which behaves pretty well). These smaller language models ain't so regular. Ok, like consider a basic thing you want to do with a classifier: understand its behavior on a held out data set. Since no one knows what is really in the training data (since its so large), its quite hard to understand what the model can generalize about and what it has just accidentally memorized. CF reports that GPT4 doesn't perform nearly as well on even simple programming exercises that are chosen in such a way as to be sure they weren't in the training data. There is enormous potential for statistical fuck ups here. Prompt engineering, for instance, is an easy place for over-fitting to happen as a prompt is fine tuned on data the prompt engineer has and thus fails to generalize to new data. I do think there is a lot of value here, but I'm also sure that sloppy use of large language models is going to cause a bunch of trouble in the short to medium term, generate a lot of garbage, pollute a lot of databases, etc, while we figure all this stuff out.
- rcme 4y agoNothing about a zero or few-shot classifier precludes you from validating on a data set.
- nathan_compton 4y agoYou can certainly hold out a validation set while you are writing your prompt, but you can't know whether the model is over fitting for your data set since you don't know what data was in the training set.
- 4y ago