3 ms·

This is a bit hard to believe that the system is decent at producing code which captures complex ideas and higher level structure when the tokens/param value is

by chaxor 3y ago

This is a bit hard to believe that the system is decent at producing code which captures complex ideas and higher level structure when the tokens/param value is >30 (it's ~200 here? ) The 'good' models (meaning having lots of 'knowledge' or 'memorization' about the dataset) typically tend to be around 2 tokens/param and models with decent generation of language with less knowledge/memorization are around 30 tokens/param. Perhaps the domain allows for this, but due to the fact that the linguistic interface on the input is still needed... It's hard to believe.

swyx 3y ago
this kind of critical thinking is exactly what replit is going to need for their stated goal of doing whole-app generation. right now they only test it on AmjadEval. you… might wanna consider joining them to work on it?
- chaxor 3y ago
  I'm not sure noticing tokens/params or simplicial modeling properties occur requires much critical thought - perhaps it's just a standard first thought for anyone given an LM now. I've worked tangentially to NLP for about 7 years in academia, but most of my work has been focused on a less known field of mathematics applied to either to outputs or to the NNs, as well as bioinformatics. As such, my expertise may not be as refined as the real players in the field such as Glaese, Finn, Velockovic, etc. but I try typically keep up with the actual key advancements in the field (usually the stuff few people notice). This area takes far too much compute capability for many people to actually become experts in it, so much of my interests have been less on large LMs and more towards algorithms. But I suppose I agree that it is frustrating to see how little knowledge many of the hype-filled crowds possess that are piling into this area. (Not calling anyone specifically out in this thread, just in general across the internet)
EvgeniyZh 3y ago
Are you saying the less you train the model the better it is? I'm confused
- swyx 3y ago
  i believe GP is referencing the Kaplan and Chinchilla scaling laws. we reference those in the podcast but i’m not sure if some deeper insight is being hinted at here where different scaling laws apply for different domains/purposes
  - EvgeniyZh 3y ago
    But these say exactly the opposite, the more tok/param the better. There is some optimum after which you need more training FLOPS to improve than if you add parameters but it is definitely not the other way around
gnramires 3y ago
Tokens/param shouldn't matter more than the total training FLOPs, I believe. Clearly if we train a your claimed 'ideal' 2 tokens/param a very small dataset (not many tokens in the first place), it wouldn't have enough data to properly learn the relevant languages. Once there is enough data, then it becomes a question of model capacity (does it have enough degrees of freedom to support the computational structures needed?). I believe the overparametrization largely helps with generalization and reducing overfitting, at 2 tokens/param there's much more degrees of freedom than structures that can be learned from what I can tell (the extra capacity just provides good breathing room for internal structures). But if your model has enough capacity, and you can find a good enough training method (and you have enough data to learn the task), then you should be able to succeed in arbitrary low tokens/param, which is good to keep in mind to make efficient models.

by chaxor 3y ago

swyx 3y ago

this kind of critical thinking is exactly what replit is going to need for their stated goal of doing whole-app generation. right now they only test it on AmjadEval. you… might wanna consider joining them to work on it?

chaxor 3y ago
I'm not sure noticing tokens/params or simplicial modeling properties occur requires much critical thought - perhaps it's just a standard first thought for anyone given an LM now. I've worked tangentially to NLP for about 7 years in academia, but most of my work has been focused on a less known field of mathematics applied to either to outputs or to the NNs, as well as bioinformatics. As such, my expertise may not be as refined as the real players in the field such as Glaese, Finn, Velockovic, etc. but I try typically keep up with the actual key advancements in the field (usually the stuff few people notice). This area takes far too much compute capability for many people to actually become experts in it, so much of my interests have been less on large LMs and more towards algorithms. But I suppose I agree that it is frustrating to see how little knowledge many of the hype-filled crowds possess that are piling into this area. (Not calling anyone specifically out in this thread, just in general across the internet)

EvgeniyZh 3y ago

Are you saying the less you train the model the better it is? I'm confused

swyx 3y ago
i believe GP is referencing the Kaplan and Chinchilla scaling laws. we reference those in the podcast but i’m not sure if some deeper insight is being hinted at here where different scaling laws apply for different domains/purposes
- EvgeniyZh 3y ago
  But these say exactly the opposite, the more tok/param the better. There is some optimum after which you need more training FLOPS to improve than if you add parameters but it is definitely not the other way around

gnramires 3y ago

Tokens/param shouldn't matter more than the total training FLOPs, I believe. Clearly if we train a your claimed 'ideal' 2 tokens/param a very small dataset (not many tokens in the first place), it wouldn't have enough data to properly learn the relevant languages. Once there is enough data, then it becomes a question of model capacity (does it have enough degrees of freedom to support the computational structures needed?). I believe the overparametrization largely helps with generalization and reducing overfitting, at 2 tokens/param there's much more degrees of freedom than structures that can be learned from what I can tell (the extra capacity just provides good breathing room for internal structures). But if your model has enough capacity, and you can find a good enough training method (and you have enough data to learn the task), then you should be able to succeed in arbitrary low tokens/param, which is good to keep in mind to make efficient models.