4 ms·

I just used Groq / llama-7b to classify 20,000 rows of Google sheets data (Sidebar archive links) that would have taken me way longer... Every one I've spot che

I just used Groq / llama-7b to classify 20,000 rows of Google sheets data (Sidebar archive links) that would have taken me way longer... Every one I've spot checked right now has been correct, and I might write another checker to scan the results just in case. Even w/ a 20% failure it's better than not having the classifications

lifeisstillgood 2y ago
Sorry this actually sounds like a real use case. What was the classification? (I tried google “sidebar archive”). I assume somehow you visited 20,000 web pages and it classified the text on the page? How was that achieved ? You ran a local llama?
- karles 2y ago
  We had ChatGPT look at 200.000 products, and make a navigation structure in 3 tiers based on the attributes of each product. The validation took 2% of the time it would have taken to manually create the hierarchy ourselves. I think that even the simple LLM's are very well suited for classification-tasks, where very little prompting is needed.
  - lifeisstillgood 2y ago
    Sorry to harp on. So you had a list of products (what sort - I am thinking like widgets from a wholesaler and you want to have a three tier menu for an e-commerce site?) I am guessing each product has a description - like from Amazon, and chatgpt read the description and said “aha this is a Television/LCD/50inch or Underwear/flimsy/bra I assume you sent in 200,000 different queries - but how did you get it to return three tiers? (Maybe I need to read one of those “become a ChatGPt expert” blogs
    - t-writescode 2y ago
      I'm not this person; but, I've been working on LLMs pretty aggressively for the last 6ish months and I have some ideas of how this __could__ be done. You could plainly ask the LLM something like this as the query goes on: "Please provide 3 categories that this product could exist under, with increasing specificity in the following format: { "broad category": "a broad category that would encompass this product, as well as others, for example 'televisions' for a 50" OLED LG with Roku integration", "category": "a narrower category that describes this product more aggressively, for example 'Smart Televisions'", "narrow category": "an even narrower category that describes this product and its direct competitors, for example OLED Smart televisions" } A next question you'll have pretty quick is, "Well, what if sometimes it returns 'Smart televisions' and other times it returns 'Smart TVs', won't that result in multiple of the same category?" And that's a good and valid question, so you then have another query that takes the categories that have been provided to you and asks for synonyms, alternative spellings, etc, such as: "Given a product categorization of a specific level of specificity, please provide a list of words and phrases that mean the same thing". In OpenAI's backend - and many of them, I think, you can have the api run the query multiple times and get back multiple answers. enumerate over those answers, build the graph, and you can have all that data in an easy to read and follow format! It might not be perfect, but it should be pretty good!
      - _lvbh 2y ago
        > Well, what if sometimes it returns 'Smart televisions' and other times it returns 'Smart TVs', won't that result in multiple of the same category Text similarity works well in this case. You can just use cosine similarity and merge ones that are very close or ask GPT to compare for those on the edge
- hobofan 2y ago
  It sounds like a real use case, but possibly quite overkill to use an LLM. Unless you need to have some "reasoning" to classify the documents correctly, a much more lightweight BERT-like model (RoBERTa or DistilBERT) will perform on par in accuracy while being a lot faster.
  - t-writescode 2y ago
    "while being a lot faster", yes; but something that LLMs do that those other tools don't is being hilariously approachable. LLMs can operate as a very, very *very* approachable natural language processing model without needing to know all the gritty details of NLP.
smusamashah 2y ago
I classified ~1000 GBA game roms files by using their file names to put each in a category folder. It worked like 90% correctly. Used GPT 3.5 and therefore it didn't adhere to my provided list of categories but they were mostly not wrong otherwise. https://gist.github.com/SMUsamaShah/20f24e80cfe962d26af5315e44669b18 https://gist.github.com/SMUsamaShah/20f24e80cfe962d26af5315e...
runeks 2y ago
> Every one I've spot checked right now has been correct, and I might write another checker to scan the results just in case. If you already have the answers to verify the LLM output against why not just use those to begin with?
- kees99 2y ago
  Not GP, but I would imagine "another checker to scan the results" would be another NN classifier. Thinking being that you'd compare outputs of the two, and under assumption of the results being statistically independent from each other and of similar quality, say 1% difference between the two in said comparison, would suggest ~ 0.5% error rate from "ground truth".
- TeMPOraL 2y ago
  Maybe their problem is using LLM to solve f:X→Y, where the validation, V:{X,Y}→{0,1}, is trivial to compute?
physicsguy 2y ago
The problem isn't that it's not useful for self driven tasks like that, it's that you can't really integrate that into a product that does task X because when someone buys a system to do task X, they want it to be more reliable than 80%.
- p1necone 2y ago
  Stick a slick UI that lets the end user quickly fix up just the bits it got wrong and flip through documents quickly and 80% correct can still be a massive timesaver.
  - kgeist 2y ago
    We're thinking about adding AI to the product and that's the path I'd like to take. View AI as an intern who can mistakes, and provide a UI where the user can review what the AI is planning to do.
    - whiplash451 2y ago
      Except that people hate monitoring an intern all day, regardless of whether it is a human or a machine.
      - deleted 2y ago
        [deleted]
      - techostritch 2y ago
        I think this is going to be a heavy lift, and one of the reasons I think a chat bot is not the right UX. Every time someone says “all you need to get to do to get ChatGPT working is provide it explicitly requirements and iterate”, and for a lot of coding tasks it’s much easier to just hack on code for a while, then be a manager to a 80% right intern.
  - physicsguy 2y ago
    I think that can kind of work for B2C things, but is much less likely to do so for B2B. Just as an example, I work on industrial maintenance software, and customers expect us to catch problems with their machinery 100% of the time, and in time to prevent it. Sometimes faults start and progress to failure faster than they're able to send data to us, but they still are upset that we didn't catch it. It doesn't matter whether that's reasonable or not, there are a lot of people who expect software systems to be totally reliable at what they do, and don't want to accept less.

lifeisstillgood 2y ago

Sorry this actually sounds like a real use case. What was the classification? (I tried google “sidebar archive”). I assume somehow you visited 20,000 web pages and it classified the text on the page? How was that achieved ? You ran a local llama?

karles 2y ago
We had ChatGPT look at 200.000 products, and make a navigation structure in 3 tiers based on the attributes of each product. The validation took 2% of the time it would have taken to manually create the hierarchy ourselves. I think that even the simple LLM's are very well suited for classification-tasks, where very little prompting is needed.
- lifeisstillgood 2y ago
  Sorry to harp on. So you had a list of products (what sort - I am thinking like widgets from a wholesaler and you want to have a three tier menu for an e-commerce site?) I am guessing each product has a description - like from Amazon, and chatgpt read the description and said “aha this is a Television/LCD/50inch or Underwear/flimsy/bra I assume you sent in 200,000 different queries - but how did you get it to return three tiers? (Maybe I need to read one of those “become a ChatGPt expert” blogs
  - t-writescode 2y ago
    I'm not this person; but, I've been working on LLMs pretty aggressively for the last 6ish months and I have some ideas of how this __could__ be done. You could plainly ask the LLM something like this as the query goes on: "Please provide 3 categories that this product could exist under, with increasing specificity in the following format: { "broad category": "a broad category that would encompass this product, as well as others, for example 'televisions' for a 50" OLED LG with Roku integration", "category": "a narrower category that describes this product more aggressively, for example 'Smart Televisions'", "narrow category": "an even narrower category that describes this product and its direct competitors, for example OLED Smart televisions" } A next question you'll have pretty quick is, "Well, what if sometimes it returns 'Smart televisions' and other times it returns 'Smart TVs', won't that result in multiple of the same category?" And that's a good and valid question, so you then have another query that takes the categories that have been provided to you and asks for synonyms, alternative spellings, etc, such as: "Given a product categorization of a specific level of specificity, please provide a list of words and phrases that mean the same thing". In OpenAI's backend - and many of them, I think, you can have the api run the query multiple times and get back multiple answers. enumerate over those answers, build the graph, and you can have all that data in an easy to read and follow format! It might not be perfect, but it should be pretty good!
    - _lvbh 2y ago
      > Well, what if sometimes it returns 'Smart televisions' and other times it returns 'Smart TVs', won't that result in multiple of the same category Text similarity works well in this case. You can just use cosine similarity and merge ones that are very close or ask GPT to compare for those on the edge
hobofan 2y ago
It sounds like a real use case, but possibly quite overkill to use an LLM. Unless you need to have some "reasoning" to classify the documents correctly, a much more lightweight BERT-like model (RoBERTa or DistilBERT) will perform on par in accuracy while being a lot faster.
- t-writescode 2y ago
  "while being a lot faster", yes; but something that LLMs do that those other tools don't is being hilariously approachable. LLMs can operate as a very, very *very* approachable natural language processing model without needing to know all the gritty details of NLP.

smusamashah 2y ago

I classified ~1000 GBA game roms files by using their file names to put each in a category folder. It worked like 90% correctly. Used GPT 3.5 and therefore it didn't adhere to my provided list of categories but they were mostly not wrong otherwise. https://gist.github.com/SMUsamaShah/20f24e80cfe962d26af5315e44669b18 https://gist.github.com/SMUsamaShah/20f24e80cfe962d26af5315e...

runeks 2y ago

> Every one I've spot checked right now has been correct, and I might write another checker to scan the results just in case. If you already have the answers to verify the LLM output against why not just use those to begin with?

kees99 2y ago
Not GP, but I would imagine "another checker to scan the results" would be another NN classifier. Thinking being that you'd compare outputs of the two, and under assumption of the results being statistically independent from each other and of similar quality, say 1% difference between the two in said comparison, would suggest ~ 0.5% error rate from "ground truth".
TeMPOraL 2y ago
Maybe their problem is using LLM to solve f:X→Y, where the validation, V:{X,Y}→{0,1}, is trivial to compute?

physicsguy 2y ago

The problem isn't that it's not useful for self driven tasks like that, it's that you can't really integrate that into a product that does task X because when someone buys a system to do task X, they want it to be more reliable than 80%.

p1necone 2y ago
Stick a slick UI that lets the end user quickly fix up just the bits it got wrong and flip through documents quickly and 80% correct can still be a massive timesaver.
- kgeist 2y ago
  We're thinking about adding AI to the product and that's the path I'd like to take. View AI as an intern who can mistakes, and provide a UI where the user can review what the AI is planning to do.
  - whiplash451 2y ago
    Except that people hate monitoring an intern all day, regardless of whether it is a human or a machine.
    - deleted 2y ago
      [deleted]
    - techostritch 2y ago
      I think this is going to be a heavy lift, and one of the reasons I think a chat bot is not the right UX. Every time someone says “all you need to get to do to get ChatGPT working is provide it explicitly requirements and iterate”, and for a lot of coding tasks it’s much easier to just hack on code for a while, then be a manager to a 80% right intern.
- physicsguy 2y ago
  I think that can kind of work for B2C things, but is much less likely to do so for B2B. Just as an example, I work on industrial maintenance software, and customers expect us to catch problems with their machinery 100% of the time, and in time to prevent it. Sometimes faults start and progress to failure faster than they're able to send data to us, but they still are upset that we didn't catch it. It doesn't matter whether that's reasonable or not, there are a lot of people who expect software systems to be totally reliable at what they do, and don't want to accept less.