3 ms·

Classification tasks come to mind

• famouswaffles 3y ago
  LLMs are better at that though. Sure you may not require them but it certainly wouldn't be for a lack of accuracy. https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 https://www.artisana.ai/articles/gpt-4-outperforms-elite-cro... https://arxiv.org/abs/2303.15056 https://arxiv.org/abs/2303.15056
  • janalsncm 3y ago
    That compares ChatGPT to mechanical turk, not to a smaller, more specialized model. Mechanical Turk is just crowdsourcing.
    • famouswaffles 3y ago
      The first one also compares GPT-4 to the researches themselves. Smaller specialized models don't beat humans at these tasks. That's why turk is used here in the first place (It's certainly not cheaper) and why GPT beating them is worthy of a paper on its own.
      • janalsncm 3y ago
        Well it really depends on the task. If it can be done with a regex, use a regex. We can’t make categorical statements about LLMs being better. It depends. You can also probably distill a large model to a smaller one while maintaining a lot of performance. DistillBert is almost as good as Bert at a fraction of the inference cost. GPT-3.5 and 4 also currently aren’t deterministic even with temperature zero, which is a nightmare for debugging.
      • syllogism 3y ago
        The gold standard they're comparing against was done by humans though. And a task-specific model trained on that data will be better at that task than GPT-4. What's definitely true is that getting decent data often takes some care, especially in how you define the task. And mechanical turk is often especially tricky to use well.