3 ms·

> dumb algorithms I’m not an expert and haven’t seen architecture of ChatGPT. But it’s probably far from dumb. You can’t simply put more data into any algorith

by f6v 4y ago

> dumb algorithms I’m not an expert and haven’t seen architecture of ChatGPT. But it’s probably far from dumb. You can’t simply put more data into any algorithm and expect improvement. You need a model to be able to take advantage of the data and train in reasonable time. That’s exactly what’s been happening in ML field.

dwaltrip 4y ago
A huge portion of the improvements in each successive version of gpt came from increasing the model size and the amount of training data. The research trends indicate that there still are additional improvements to be found by continuing to scale things up, especially when it comes to the amount of training data. Not indefinitely, obviously. But we haven’t exhausted the available data sources yet, and the curves for prediction error vs. amount of training data strongly imply using this additional data would result in increased performance. Do you have specific reasons for thinking that gpt-4 is near the end of this trend? I haven’t seen any compelling arguments for this.
- f6v 4y ago
  > Do you have specific reasons for thinking that gpt-4 is near the end of this trend? I haven’t said that. I pointed that you can’t simply put more data into it and expect improvements. There’s definitely a lot of engineering done on the models to make them capable of using all the data. I can give you an example. You can’t simply add more convolutional layers to a CNN and feed more images. The architectures of image models include many tricks, like inception layers. I suspect there’re similar challenges in LLMs. And I’m not sure whether current ChatGPT can use more data to become better at reasoning, or at some point they’d need to come up with different architectures.
  - whimsicalism 4y ago
    You are largely wrong, to be honest. Mostly, you can simply stack more transformers and (as long as you have layer norm), it can train on more data (of fixed sequence length). There are some tricks & tips, but the vast majority lie outside of the architecture improvements.
    - f6v 4y ago
      I’m not an expert, but I assume there’s a limit where «simply adding stuff” stops to provide improvements when you take training time into account. And my gut feeling is that true reasoning requires different architectures.

by f6v 4y ago

dwaltrip 4y ago

A huge portion of the improvements in each successive version of gpt came from increasing the model size and the amount of training data. The research trends indicate that there still are additional improvements to be found by continuing to scale things up, especially when it comes to the amount of training data. Not indefinitely, obviously. But we haven’t exhausted the available data sources yet, and the curves for prediction error vs. amount of training data strongly imply using this additional data would result in increased performance. Do you have specific reasons for thinking that gpt-4 is near the end of this trend? I haven’t seen any compelling arguments for this.

f6v 4y ago
> Do you have specific reasons for thinking that gpt-4 is near the end of this trend? I haven’t said that. I pointed that you can’t simply put more data into it and expect improvements. There’s definitely a lot of engineering done on the models to make them capable of using all the data. I can give you an example. You can’t simply add more convolutional layers to a CNN and feed more images. The architectures of image models include many tricks, like inception layers. I suspect there’re similar challenges in LLMs. And I’m not sure whether current ChatGPT can use more data to become better at reasoning, or at some point they’d need to come up with different architectures.
- whimsicalism 4y ago
  You are largely wrong, to be honest. Mostly, you can simply stack more transformers and (as long as you have layer norm), it can train on more data (of fixed sequence length). There are some tricks & tips, but the vast majority lie outside of the architecture improvements.
  - f6v 4y ago
    I’m not an expert, but I assume there’s a limit where «simply adding stuff” stops to provide improvements when you take training time into account. And my gut feeling is that true reasoning requires different architectures.