5 ms·

I think the Bitter Lessons do not consider two important points with regards to neural nets : 1. Not all nets work for all problems, those that work tend to ha

I think the Bitter Lessons do not consider two important points with regards to neural nets : 1. Not all nets work for all problems, those that work tend to have the right inductive biases. We discovered the architectures partially by trial an error, nevertheless they work because of encoded prior information. 2. Data and computation are bounded. GPT4 was basically trained on all text, further advancements probably need more insight not more data.

psychoslave 3y ago
What do you mean with more insights? If they are not represented as some data, how could they ever influence any computational model?
- davegirdywww 3y ago
  Nobody said anything about insights.
- niemandhier 3y ago
  By building the model in such a way that assumptions about the data are in the model structure. ConvNets hardcode translation equivariance, more general convolutions can hardcode more general equivariant structures. The whole field of geometric deep learning is about constructing nets based on insights about the structure of the data manifold, and many successful nets turn out to do this.
the8472 3y ago
> GPT4 was basically trained on all text Well, there's multi-modal training. There's tons of untapped audiovisual data.
- reaperman 3y ago
  GPT4 is a multi-modal model. They haven't exposed a way to use the image embeddings, so consumers can't utilize it, but the model accepts image input and it was trained on images and text. Yes, there are other modals that can be incorporated, and reinforcement learning is still pretty nascent/very much unsolved.
  - the8472 3y ago
    The fact that it's 1) multi-modal and 2) we're approaching text exhaustion does not imply 3) that other modalities are exhausted
    - reaperman 3y ago
      Yeah I just like pointing out that GPT-4 is multimodal because most people don't seem to realize this, as they never read the GPT-4 papers.
famouswaffles 3y ago
The essence of the bitter lesson is that the less inductive bias we try to bake in, the better performing they tend to be. Transformers are preferred to ConvNets these days in Computer Vision despite the latter having all sorts of vision based inductive biases. GPT-4 has not been trained on all text lol
- niemandhier 3y ago
  1. Partially that just moves where we must search for implicitly included symmetries. Word embedding most famously create structure that allows to define shift operations in the embedding space “doctor + female = nurse” might be the most infamous. By evolution only those word embedding so that yield good results are used. Again just because we did not put the structure there, but discovered it by trial and error does not mean that the structure is not key to the success. 2. Gpt 4 was trained on 13T tokens, all books ever written would be about 6.5 T tokens by Fermi calculation.
  - famouswaffles 3y ago
    1. Sorry but that's still far less bias than ConvNets for vision. And far less bias than previous methods for language. The less bias the better. It's simple. 2. Assuming the rumor is fact, GPT was trained for multiple epochs and books are a small percent of what trains LLMs lol.

psychoslave 3y ago

What do you mean with more insights? If they are not represented as some data, how could they ever influence any computational model?

davegirdywww 3y ago
Nobody said anything about insights.
niemandhier 3y ago
By building the model in such a way that assumptions about the data are in the model structure. ConvNets hardcode translation equivariance, more general convolutions can hardcode more general equivariant structures. The whole field of geometric deep learning is about constructing nets based on insights about the structure of the data manifold, and many successful nets turn out to do this.

the8472 3y ago

> GPT4 was basically trained on all text Well, there's multi-modal training. There's tons of untapped audiovisual data.

reaperman 3y ago
GPT4 is a multi-modal model. They haven't exposed a way to use the image embeddings, so consumers can't utilize it, but the model accepts image input and it was trained on images and text. Yes, there are other modals that can be incorporated, and reinforcement learning is still pretty nascent/very much unsolved.
- the8472 3y ago
  The fact that it's 1) multi-modal and 2) we're approaching text exhaustion does not imply 3) that other modalities are exhausted
  - reaperman 3y ago
    Yeah I just like pointing out that GPT-4 is multimodal because most people don't seem to realize this, as they never read the GPT-4 papers.

famouswaffles 3y ago

The essence of the bitter lesson is that the less inductive bias we try to bake in, the better performing they tend to be. Transformers are preferred to ConvNets these days in Computer Vision despite the latter having all sorts of vision based inductive biases. GPT-4 has not been trained on all text lol

niemandhier 3y ago
1. Partially that just moves where we must search for implicitly included symmetries. Word embedding most famously create structure that allows to define shift operations in the embedding space “doctor + female = nurse” might be the most infamous. By evolution only those word embedding so that yield good results are used. Again just because we did not put the structure there, but discovered it by trial and error does not mean that the structure is not key to the success. 2. Gpt 4 was trained on 13T tokens, all books ever written would be about 6.5 T tokens by Fermi calculation.
- famouswaffles 3y ago
  1. Sorry but that's still far less bias than ConvNets for vision. And far less bias than previous methods for language. The less bias the better. It's simple. 2. Assuming the rumor is fact, GPT was trained for multiple epochs and books are a small percent of what trains LLMs lol.