3 ms·

If they need 100x the available training data to scale beyond their current capabilities they clearly don't scale particularly well.

famouswaffles 3y ago
Good thing they don't then
- geysersam 3y ago
  I don't understand what you mean. To improve performance using the same basic architecture the model need to scale both compute and training data. Where are we going to find another 10x the web sized text training corpus?
  - famouswaffles 3y ago
    Gpt-4 was allegedly trained on 6T tokens for 2 epochs (i.e twice). 6T is far from exhausting the text data we have. Here's a 5 language 30T token dataset from web scrapped data. https://github.com/togethercomputer/RedPajama-Data https://github.com/togethercomputer/RedPajama-Data And that is just web scrapped data. There are trillions of valuable tokens worth of text from the likes of pdfs/ebooks, academic journals and other documents that essentially has no web presence otherwise. https://annas-archive.org/llm https://annas-archive.org/llm >To improve performance using the same basic architecture the model need to scale both compute and training data. What you are trying to say here is that you need to scale both parameters and data as scaling data increases compute. That said, it's not really true. It would be best if you scaled both data and parameters but you can get increased performance just scaling one or the other.
    - geysersam 3y ago
      The 6T token dataset is surely a high quality subset / refined extract from much larger public datasets. It's misleading to compare their sizes directly.
      - famouswaffles 3y ago
        We don't know what the dataset is. "high quality subset from much larger public datasets" is not just inherently speculation, it's flat out wrong as no such public datasets existed when GPT-4 was trained.
exe34 3y ago
Do you know much about how this all works? I've been thinking about how a human child ingests (maybe) petabytes of tokens every day through video and action. They initiate movement and observe the consequences through video, audio, proprioception, etc. The decades of schooling afterwards seems like fine tuning on a base model.
- cma 3y ago
  The base model is likely to have lots of genetic determined stuff that is there before any training: some animals like horses can walk and perceive/navigate very soon after being born. There's also an innate fear of snake shapes in most mammals.
  - exe34 3y ago
    Another one that's bothered me is how our models so often start from scratch (aside from fine tuning or transfer learning) - I wish we could just start with say a smaller model with low resolution and grow it into a higher resolution one, a small LLM that can be grown into different specialisations (which we do with fine tuning - but do they all need the same number of layers, size of each layer, etc).

famouswaffles 3y ago

Good thing they don't then

geysersam 3y ago
I don't understand what you mean. To improve performance using the same basic architecture the model need to scale both compute and training data. Where are we going to find another 10x the web sized text training corpus?
- famouswaffles 3y ago
  Gpt-4 was allegedly trained on 6T tokens for 2 epochs (i.e twice). 6T is far from exhausting the text data we have. Here's a 5 language 30T token dataset from web scrapped data. https://github.com/togethercomputer/RedPajama-Data https://github.com/togethercomputer/RedPajama-Data And that is just web scrapped data. There are trillions of valuable tokens worth of text from the likes of pdfs/ebooks, academic journals and other documents that essentially has no web presence otherwise. https://annas-archive.org/llm https://annas-archive.org/llm >To improve performance using the same basic architecture the model need to scale both compute and training data. What you are trying to say here is that you need to scale both parameters and data as scaling data increases compute. That said, it's not really true. It would be best if you scaled both data and parameters but you can get increased performance just scaling one or the other.
  - geysersam 3y ago
    The 6T token dataset is surely a high quality subset / refined extract from much larger public datasets. It's misleading to compare their sizes directly.
    - famouswaffles 3y ago
      We don't know what the dataset is. "high quality subset from much larger public datasets" is not just inherently speculation, it's flat out wrong as no such public datasets existed when GPT-4 was trained.

exe34 3y ago

Do you know much about how this all works? I've been thinking about how a human child ingests (maybe) petabytes of tokens every day through video and action. They initiate movement and observe the consequences through video, audio, proprioception, etc. The decades of schooling afterwards seems like fine tuning on a base model.

cma 3y ago
The base model is likely to have lots of genetic determined stuff that is there before any training: some animals like horses can walk and perceive/navigate very soon after being born. There's also an innate fear of snake shapes in most mammals.
- exe34 3y ago
  Another one that's bothered me is how our models so often start from scratch (aside from fine tuning or transfer learning) - I wish we could just start with say a smaller model with low resolution and grow it into a higher resolution one, a small LLM that can be grown into different specialisations (which we do with fine tuning - but do they all need the same number of layers, size of each layer, etc).