4 ms·

Even if LLMs didn't advance at all from this point onward, there's still loads of productive work that could be optimized / fully automated by them, at no worse

Even if LLMs didn't advance at all from this point onward, there's still loads of productive work that could be optimized / fully automated by them, at no worse output quality than the low-skilled humans we're currently throwing at that work.

SchemaLoad 9mo ago
How much of the current usage is productive work that's worth paying for vs personal usage / spam that would just drop off after usage charges come in? I imagine flooding youtube and instagram with slop videos would reduce if users had to pay fair prices to use the models. The companies might also downgrade the quality of the models to make it more viable to provide as an ad supported service which would again reduce utilisation.
- unsupp0rted 9mo ago
  For any "click here and type into a box" job for which you'd hire a low-skilled worker and give them an SOP to follow, you can have an LLM-ish tool do it. And probably for the slightly more skilled email jobs that have infiltrated nearly all companies too. Is that productive work? Well if people are getting paid, often a multiple of minimum wage, then it's productive-seeming enough.
  - greree 9mo ago
    Another bozo making fun of other job classes. Why are there still customer service reps? Shouldn’t they all be gone by now due to this amazing technology? Ah, tumbleweed.
- bethekidyouwant 9mo ago
  Who is generating videos for free?
pvab3 9mo ago
inference requires a fraction of the power that training does. According to the Villalobos paper, the median date is 2028. At some point we won't be training bigger and bigger models every month. We will run out of additional material to train on, things will continue commodifying, and then the amount of training happening will significantly decrease unless new avenues open for new types of models. But our current LLMs are much more compute-intensive than any other type of generative or task-specific model
- zozbot234 9mo ago
  > We will run out of additional material to train on This sounds a bit silly. More training will generally result in better modeling, even for a fixed amount of genuine original data. At current model sizes, it's essentially impossible to overfit to the training data so there's no reason why we should just "stop".
  - pvab3 9mo ago
    I'm just talking about text generated by human beings. You can keep retraining with more parameters on the same corpus https://proceedings.mlr.press/v235/villalobos24a.html https://proceedings.mlr.press/v235/villalobos24a.html
    - x-complexity 9mo ago
      > I'm just talking about text generated by human beings. That in itself is a goalpost shift from > > We will run out of additional material to train on Where it is implied "additional material" === "all data, human + synthetic" ------ There's still some headroom left in the synthetic data playground, as cited in the paper linked: https://proceedings.mlr.press/v235/villalobos24a.html https://proceedings.mlr.press/v235/villalobos24a.html ( https://openreview.net/pdf?id=ViZcgDQjyG https://openreview.net/pdf?id=ViZcgDQjyG ) "On the other hand, training on synthetic data has shown much promise in domains where model outputs are relatively easy to verify, such as mathematics, programming, and games (Yang et al., 2023; Liu et al., 2023; Haluptzok et al., 2023)." With the caveat that translating this success outside of these domains is hit-or-miss: "What is less clear is whether the usefulness of synthetic data will generalize to domains where output verification is more challenging, such as natural language." The main bottleneck for this area of the woods will be (X := how many additional domains can be made easily verifiable). So long as (the rate of X) >> (training absorption rate), the road can be extended for a while longer.

SchemaLoad 9mo ago

How much of the current usage is productive work that's worth paying for vs personal usage / spam that would just drop off after usage charges come in? I imagine flooding youtube and instagram with slop videos would reduce if users had to pay fair prices to use the models. The companies might also downgrade the quality of the models to make it more viable to provide as an ad supported service which would again reduce utilisation.

unsupp0rted 9mo ago
For any "click here and type into a box" job for which you'd hire a low-skilled worker and give them an SOP to follow, you can have an LLM-ish tool do it. And probably for the slightly more skilled email jobs that have infiltrated nearly all companies too. Is that productive work? Well if people are getting paid, often a multiple of minimum wage, then it's productive-seeming enough.
- greree 9mo ago
  Another bozo making fun of other job classes. Why are there still customer service reps? Shouldn’t they all be gone by now due to this amazing technology? Ah, tumbleweed.
bethekidyouwant 9mo ago
Who is generating videos for free?

pvab3 9mo ago

inference requires a fraction of the power that training does. According to the Villalobos paper, the median date is 2028. At some point we won't be training bigger and bigger models every month. We will run out of additional material to train on, things will continue commodifying, and then the amount of training happening will significantly decrease unless new avenues open for new types of models. But our current LLMs are much more compute-intensive than any other type of generative or task-specific model

zozbot234 9mo ago
> We will run out of additional material to train on This sounds a bit silly. More training will generally result in better modeling, even for a fixed amount of genuine original data. At current model sizes, it's essentially impossible to overfit to the training data so there's no reason why we should just "stop".
- pvab3 9mo ago
  I'm just talking about text generated by human beings. You can keep retraining with more parameters on the same corpus https://proceedings.mlr.press/v235/villalobos24a.html https://proceedings.mlr.press/v235/villalobos24a.html
  - x-complexity 9mo ago
    > I'm just talking about text generated by human beings. That in itself is a goalpost shift from > > We will run out of additional material to train on Where it is implied "additional material" === "all data, human + synthetic" ------ There's still some headroom left in the synthetic data playground, as cited in the paper linked: https://proceedings.mlr.press/v235/villalobos24a.html https://proceedings.mlr.press/v235/villalobos24a.html ( https://openreview.net/pdf?id=ViZcgDQjyG https://openreview.net/pdf?id=ViZcgDQjyG ) "On the other hand, training on synthetic data has shown much promise in domains where model outputs are relatively easy to verify, such as mathematics, programming, and games (Yang et al., 2023; Liu et al., 2023; Haluptzok et al., 2023)." With the caveat that translating this success outside of these domains is hit-or-miss: "What is less clear is whether the usefulness of synthetic data will generalize to domains where output verification is more challenging, such as natural language." The main bottleneck for this area of the woods will be (X := how many additional domains can be made easily verifiable). So long as (the rate of X) >> (training absorption rate), the road can be extended for a while longer.