5 ms·

Where's the comparison with traditional ML? In the article I only see the good things about using LLM, but there's no mention to traditional ML besides from the

Where's the comparison with traditional ML? In the article I only see the good things about using LLM, but there's no mention to traditional ML besides from the title. It would be nice to see how compares this "complex" approach against a "simple" TF-IDF + RF or SVM.

jonathankoren 3y ago
Yeah, I also find the lack of the comparison suspicious. As is the talk about “hallucinated class labels” being “helpful”. If I had to take a guess, I suspect the LLM might perform a touch better, but we’re taking fractional percent better. Which is fine, if you have the volume, but a wash otherwise
- famouswaffles 3y ago
  https://news.ycombinator.com/item?id=36685921 https://news.ycombinator.com/item?id=36685921
specproc 3y ago
Yeah, my thoughts exactly. If you're running 500k in tokens through through someone else's hallucination-prone computer and paying for the privilege, I want to know why that's any better than something like SetFit. All I saw were attempts to reproduce some chatgpt output.
- espe 3y ago
  +1 for setfit. a baseline that's hard to beat.
- hellovai 3y ago
  SetFit is fairly good, and we do help train SetFit like models for the results you get, however, the issue with SetFit is that its latency and cost benefits come at the price of flexibility. If you want to add a new class, to update an existing one, it requires training a new model. Sometimes this is ok and sometimes it's not. This is why we generally prefer a hybrid approach where some classes are using traditional models (BERT based) while others are determined by the LLM.
  - specproc 3y ago
    I guess use case is everything. There are numerous reasons, not least of which confidentiality, why chatgpt is a no go for me. What I'd like to see more of is systematic comparison between chatgpt and classic models. I was hoping to see a bit of this in this article and was disappointed.
    - hellovai 3y ago
      I appreciate the feedback, and also agree that chatgpt is a no-go for many use cases. We're working on putting together a better comparison specifically along the lines of accuracy between the LLMs (chatgpt, bard, falcon) and also traditional models. Hope that one hits the spot for you! Are their specific metrics you think might be interesting? We were primarily looking at f1/accuracy for this task, but also attempting to see what types of classes they work well in using semantic similarity.
- famouswaffles 3y ago
  https://news.ycombinator.com/item?id=36685921 https://news.ycombinator.com/item?id=36685921
hellovai 3y ago
Thanks Alex, in this article we focused more on deployment comparisons, for example the cost and latency of what it would take to deploy a BERT based model vs LLMs. In a future article, we're planning on posting accuracy comparisons as well, but here we want to evaluate a few other architectures for comparison. For example, at 1TPS with 1k tokens, chat-gpt-turbo would cost almost $5k vs a simpler BERT model you could run for under $50. This is probably very obvious to some people, but a lot of people's first experience with any sort of AI is often an LLM, so this is just the first of many posts we hope to share.
viraptor 3y ago
Or even slightly fancy Word2vec/USE or even sentence transformers with clustering that you can trivially run locally rather than a full blown conversational LLM. I'd love to see a large scale comparison.
famouswaffles 3y ago
Current State of the art (GPT-4) is mostly on par with experts and much better than crowdworkers. Might be overkill though. https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 https://www.artisana.ai/articles/gpt-4-outperforms-elite-cro... 3.5 (what is used here) is better than crowd workers https://arxiv.org/abs/2303.15056 https://arxiv.org/abs/2303.15056
- lisasays 3y ago
  On par with "experts", no. Per the article: "outperformed the most skilled crowdworkers" on nuanced (but not highly technical) tasks like sentiment labeling. By definition, it can't outperform the expert ensemble because that's where the gold labels come from.
  - famouswaffles 3y ago
    It's as good or better than experts on 7/18 of those benchmarks. On an additional 4, it's close (within 0.05). >By definition, it can't outperform the expert ensemble because that's where the gold labels come from. The ensemble no but it can outperform an expert trying to solve it. But yes the benchmarks are biased to the experts here.
    - lisasays 3y ago
      Thanks, I confess to skimming and missed the individual 'expert' column (presumably the mean of the individual expert scores). That said -- it looks like not only does model+ do worse than experts on the other 12/18 (not 11/18 by my counting), but when it does, it does so by a significantly wider margin (2x-3x on average). For example, the maximum model+ outperformance is on label 'Stealing' (0.11) while there are 6 labels for which the expert outperforms (by margins ranging from 0.12 to 0.29). In other words: distinctly sub-par compared with the average expert. Which is probably why they didn't claim it as a result in the paper :)
      - famouswaffles 3y ago

jonathankoren 3y ago

Yeah, I also find the lack of the comparison suspicious. As is the talk about “hallucinated class labels” being “helpful”. If I had to take a guess, I suspect the LLM might perform a touch better, but we’re taking fractional percent better. Which is fine, if you have the volume, but a wash otherwise

famouswaffles 3y ago
https://news.ycombinator.com/item?id=36685921 https://news.ycombinator.com/item?id=36685921

specproc 3y ago

Yeah, my thoughts exactly. If you're running 500k in tokens through through someone else's hallucination-prone computer and paying for the privilege, I want to know why that's any better than something like SetFit. All I saw were attempts to reproduce some chatgpt output.

espe 3y ago
+1 for setfit. a baseline that's hard to beat.
hellovai 3y ago
SetFit is fairly good, and we do help train SetFit like models for the results you get, however, the issue with SetFit is that its latency and cost benefits come at the price of flexibility. If you want to add a new class, to update an existing one, it requires training a new model. Sometimes this is ok and sometimes it's not. This is why we generally prefer a hybrid approach where some classes are using traditional models (BERT based) while others are determined by the LLM.
- specproc 3y ago
  I guess use case is everything. There are numerous reasons, not least of which confidentiality, why chatgpt is a no go for me. What I'd like to see more of is systematic comparison between chatgpt and classic models. I was hoping to see a bit of this in this article and was disappointed.
  - hellovai 3y ago
    I appreciate the feedback, and also agree that chatgpt is a no-go for many use cases. We're working on putting together a better comparison specifically along the lines of accuracy between the LLMs (chatgpt, bard, falcon) and also traditional models. Hope that one hits the spot for you! Are their specific metrics you think might be interesting? We were primarily looking at f1/accuracy for this task, but also attempting to see what types of classes they work well in using semantic similarity.
famouswaffles 3y ago
https://news.ycombinator.com/item?id=36685921 https://news.ycombinator.com/item?id=36685921

hellovai 3y ago

Thanks Alex, in this article we focused more on deployment comparisons, for example the cost and latency of what it would take to deploy a BERT based model vs LLMs. In a future article, we're planning on posting accuracy comparisons as well, but here we want to evaluate a few other architectures for comparison. For example, at 1TPS with 1k tokens, chat-gpt-turbo would cost almost $5k vs a simpler BERT model you could run for under $50. This is probably very obvious to some people, but a lot of people's first experience with any sort of AI is often an LLM, so this is just the first of many posts we hope to share.

viraptor 3y ago

Or even slightly fancy Word2vec/USE or even sentence transformers with clustering that you can trivially run locally rather than a full blown conversational LLM. I'd love to see a large scale comparison.

famouswaffles 3y ago

Current State of the art (GPT-4) is mostly on par with experts and much better than crowdworkers. Might be overkill though. https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 https://www.artisana.ai/articles/gpt-4-outperforms-elite-cro... 3.5 (what is used here) is better than crowd workers https://arxiv.org/abs/2303.15056 https://arxiv.org/abs/2303.15056

lisasays 3y ago
On par with "experts", no. Per the article: "outperformed the most skilled crowdworkers" on nuanced (but not highly technical) tasks like sentiment labeling. By definition, it can't outperform the expert ensemble because that's where the gold labels come from.
- famouswaffles 3y ago
  It's as good or better than experts on 7/18 of those benchmarks. On an additional 4, it's close (within 0.05). >By definition, it can't outperform the expert ensemble because that's where the gold labels come from. The ensemble no but it can outperform an expert trying to solve it. But yes the benchmarks are biased to the experts here.
  - lisasays 3y ago
    Thanks, I confess to skimming and missed the individual 'expert' column (presumably the mean of the individual expert scores). That said -- it looks like not only does model+ do worse than experts on the other 12/18 (not 11/18 by my counting), but when it does, it does so by a significantly wider margin (2x-3x on average). For example, the maximum model+ outperformance is on label 'Stealing' (0.11) while there are 6 labels for which the expert outperforms (by margins ranging from 0.12 to 0.29). In other words: distinctly sub-par compared with the average expert. Which is probably why they didn't claim it as a result in the paper :)
    - famouswaffles 3y ago