5 ms·

And how can one be so sure of that? Seems to me that performance is converging and we might not see a significant jump until we have another breakthrough.

lionkor 2y ago
[flagged]
scarmig 2y ago
Yeah. There are lots of things we can do with existing capabilities, but in terms of progressing beyond them all of the frontier models seem like they're a hair's breadth from each other. That is not what one would predict if LLMs had a much higher ceiling than we are currently at. I'll reserve judgment until we see GPT5, but if it becomes just a matter of who best can monetize existing capabilities, OAI isn't the best positioned.
- andrepd 2y ago
  Exactly. People like to point at the start of a logistic curve and go "behold! an exponential"
diego_sandoval 2y ago
> Seems to me that performance is converging It doesn't seem that way to me. But even if it did, video generation also seemed kind of stagnant before Sora. In general, I think The Bitter Lesson is the biggest factor at play here, and compute power is not stagnating.
- deleted 2y ago
  [deleted]
- drawnwren 2y ago
  Computer power is not stagnating, but the availability of training data is. It's not like there's a second stackoverflow or reddit to scrape.
  - robwwilliams 2y ago
    No: soon the wide wild world itself becomes training data. And for much more than just an LLM. LLM plus reinforcement learning—this is were the capacity of our in silico children will engender much parental anxiety.
    - diego_sandoval 2y ago
      Agree. However, I think the most cost-effective way to train for real world is to train in a simulated physical world first. I would assume that Boston Dynamics does exactly that, and I would expect integrated vision-action-language models to first be trained that way too.
      - pixl97 2y ago
        That's how everyone in robotics is doing these days. You take a bunch of mo-cap data and simulate it with your robot body. Then as much testing as you can with the robot and feed the behavior back in to the model for fine tuning. Unitree gives an example of the simulation versus what the robot can do in their latest video https://www.youtube.com/watch?v=GzX1qOIO1bE https://www.youtube.com/watch?v=GzX1qOIO1bE
    - Animats 2y ago
      This may create a market for surveillance camera data and phone calls. "This conversation may be recorded and used for training purposes" now takes on a new meaning. Can car makers sell info from everything that happens in their cars?
      - abenga 2y ago
        Well, this is a massively horrifying possibility.
  - bigyikes 2y ago
    It isn’t clear that we are running out of training data, and it is becoming increasingly clear that AI-generated training data actually works. For the skeptical, consider that humans can be trained on material created by less intelligent humans.
- wavemode 2y ago
  > video generation also seemed kind of stagnant before Sora I take the opposite view. I don't think video generation was stagnating at all, and was in fact probably the area of generative AI that was seeing the biggest active strides. I'm highly optimistic about the future trajectory of image and video models. By contrast, text generation has not improved significantly, in my opinion, for more than a year now, and even the improvement we saw back then was relatively marginal compared to GPT-3.5 (that is, for most day-to-day use cases we didn't really go from "this model can't do this task" to "this model can now do this task". It was more just "this model does these pre-existing tasks, in somewhat more detail".) If OpenAI really is secretly cooking up some huge reasoning improvements for their text models, I'll eat my hat. But for now I'm skeptical.
  - Eisenstein 2y ago
    > By contrast, text generation has not improved significantly, in my opinion, for more than a year now With less than $800 worth of hardware including everything but the monitor, you can run an open weight model more powerful than GPT 3.5 locally, at around 6 - 7T/s[0]. I would say that is a huge improvement. [0] https://www.reddit.com/r/LocalLLaMA/comments/1cmmob0/p40_build_specs_and_benchmark_data_for_anyone/ https://www.reddit.com/r/LocalLLaMA/comments/1cmmob0/p40_bui...
aantix 2y ago
The use of AI in the research of AI accelerates everything.
- thefaux 2y ago
  I'm not sure of this. The jury is still out on most ai tools. Even if it is true, it may be in a kind of strange reverse way: people innovating by asking what ai can't do and directing their attention there.
  - bigyikes 2y ago
    There is an increasing amount of evidence that using AI to train other AI is a viable path forward. E.g. using LLMs to generate training data or tune RL policies
- jcd000 2y ago
  I bet this will also cause model regressions.

lionkor 2y ago

[flagged]

scarmig 2y ago

Yeah. There are lots of things we can do with existing capabilities, but in terms of progressing beyond them all of the frontier models seem like they're a hair's breadth from each other. That is not what one would predict if LLMs had a much higher ceiling than we are currently at. I'll reserve judgment until we see GPT5, but if it becomes just a matter of who best can monetize existing capabilities, OAI isn't the best positioned.

andrepd 2y ago
Exactly. People like to point at the start of a logistic curve and go "behold! an exponential"

diego_sandoval 2y ago

> Seems to me that performance is converging It doesn't seem that way to me. But even if it did, video generation also seemed kind of stagnant before Sora. In general, I think The Bitter Lesson is the biggest factor at play here, and compute power is not stagnating.

deleted 2y ago
[deleted]
drawnwren 2y ago
Computer power is not stagnating, but the availability of training data is. It's not like there's a second stackoverflow or reddit to scrape.
- robwwilliams 2y ago
  No: soon the wide wild world itself becomes training data. And for much more than just an LLM. LLM plus reinforcement learning—this is were the capacity of our in silico children will engender much parental anxiety.
  - diego_sandoval 2y ago
    Agree. However, I think the most cost-effective way to train for real world is to train in a simulated physical world first. I would assume that Boston Dynamics does exactly that, and I would expect integrated vision-action-language models to first be trained that way too.
    - pixl97 2y ago
      That's how everyone in robotics is doing these days. You take a bunch of mo-cap data and simulate it with your robot body. Then as much testing as you can with the robot and feed the behavior back in to the model for fine tuning. Unitree gives an example of the simulation versus what the robot can do in their latest video https://www.youtube.com/watch?v=GzX1qOIO1bE https://www.youtube.com/watch?v=GzX1qOIO1bE
  - Animats 2y ago
    This may create a market for surveillance camera data and phone calls. "This conversation may be recorded and used for training purposes" now takes on a new meaning. Can car makers sell info from everything that happens in their cars?
    - abenga 2y ago
      Well, this is a massively horrifying possibility.
- bigyikes 2y ago
  It isn’t clear that we are running out of training data, and it is becoming increasingly clear that AI-generated training data actually works. For the skeptical, consider that humans can be trained on material created by less intelligent humans.
wavemode 2y ago
> video generation also seemed kind of stagnant before Sora I take the opposite view. I don't think video generation was stagnating at all, and was in fact probably the area of generative AI that was seeing the biggest active strides. I'm highly optimistic about the future trajectory of image and video models. By contrast, text generation has not improved significantly, in my opinion, for more than a year now, and even the improvement we saw back then was relatively marginal compared to GPT-3.5 (that is, for most day-to-day use cases we didn't really go from "this model can't do this task" to "this model can now do this task". It was more just "this model does these pre-existing tasks, in somewhat more detail".) If OpenAI really is secretly cooking up some huge reasoning improvements for their text models, I'll eat my hat. But for now I'm skeptical.
- Eisenstein 2y ago
  > By contrast, text generation has not improved significantly, in my opinion, for more than a year now With less than $800 worth of hardware including everything but the monitor, you can run an open weight model more powerful than GPT 3.5 locally, at around 6 - 7T/s[0]. I would say that is a huge improvement. [0] https://www.reddit.com/r/LocalLLaMA/comments/1cmmob0/p40_build_specs_and_benchmark_data_for_anyone/ https://www.reddit.com/r/LocalLLaMA/comments/1cmmob0/p40_bui...

aantix 2y ago

The use of AI in the research of AI accelerates everything.

thefaux 2y ago
I'm not sure of this. The jury is still out on most ai tools. Even if it is true, it may be in a kind of strange reverse way: people innovating by asking what ai can't do and directing their attention there.
- bigyikes 2y ago
  There is an increasing amount of evidence that using AI to train other AI is a viable path forward. E.g. using LLMs to generate training data or tune RL policies
jcd000 2y ago
I bet this will also cause model regressions.