5 ms·

Also see A Better Lesson: https://rodneybrooks.com/a-better-lesson/ https://rodneybrooks.com/a-better-lesson/

by v9v 3y ago

PoignardAzur 3y ago
Ehh, I'm more on Sutton's side so far. Brook's post goes over the classics (Moore's law is ending, curating a dataset requires human intervention, etc) and posits that making a huge model won't be a competitive strategy for long because it gets to expensive to train and use. It's a bit early to tell, but so far that hasn't materialized. OpenAI got state-of-the-art results with GPT-4, AFAIK by sticking for very-super-big models together. Open source experiments with LLAMA show you can still get good results with heavy quantization. Distillation hasn't be too explored by mainstream projects, but I bet there's lots of potential there too. Right now the winning strategy looks to be "go really big, then figure out how to go small".
- PartiallyTyped 3y ago
  Going really big and figuring out how to go small is also working; see Lottery Tickets, ensembles, distillation, and so on.
  - tysam_and 3y ago
    As someone who holds a speed WR with a very tiny model (<~8.5 MB or so), I would argue kindly that Sutton's lesson is about methods that scale -- and scale goes in the opposite direction too! If you do not have solid scaling, your link between micro methods predicting mega methods is broken. Hence, Sutton's bitter law forms the foundation for a few other lemmas that I think underpin what makes really effective research (which is iteration time, and how we effectively reduce it as much as possible and make it as accessible as possible -- which thankfully for ML algorithms seems to go hand in hand! <3 :')))) )
papercurt 3y ago
I think the convergence to transformer architectures already proofs this article from 2019 wrong in some ways. Even though CNNs did and do a good job, they will most likely be superseded by vision transformers. Of course you could argue that stable diffusion models and LLMs also have some domain knowledge inside. But it might be a little less with multimodal networks. So I guess the lesson should be: Don't put more domain knowledge inside than what's needed to make a product out of it right now, because everything else won't progress the field.

Also see A Better Lesson: https://rodneybrooks.com/a-better-lesson/ https://rodneybrooks.com/a-better-lesson/

by v9v 3y ago

Also see A Better Lesson: https://rodneybrooks.com/a-better-lesson/ https://rodneybrooks.com/a-better-lesson/

Ehh, I'm more on Sutton's side so far. Brook's post goes over the classics (Moore's law is ending, curating a dataset requires human intervention, etc) and posits that making a huge model won't be a competitive strategy for long because it gets to expensive to train and use. It's a bit early to tell, but so far that hasn't materialized. OpenAI got state-of-the-art results with GPT-4, AFAIK by sticking for very-super-big models together. Open source experiments with LLAMA show you can still get good results with heavy quantization. Distillation hasn't be too explored by mainstream projects, but I bet there's lots of potential there too. Right now the winning strategy looks to be "go really big, then figure out how to go small".

PartiallyTyped 3y ago
Going really big and figuring out how to go small is also working; see Lottery Tickets, ensembles, distillation, and so on.
- tysam_and 3y ago
  As someone who holds a speed WR with a very tiny model (<~8.5 MB or so), I would argue kindly that Sutton's lesson is about methods that scale -- and scale goes in the opposite direction too! If you do not have solid scaling, your link between micro methods predicting mega methods is broken. Hence, Sutton's bitter law forms the foundation for a few other lemmas that I think underpin what makes really effective research (which is iteration time, and how we effectively reduce it as much as possible and make it as accessible as possible -- which thankfully for ML algorithms seems to go hand in hand! <3 :')))) )

papercurt 3y ago

I think the convergence to transformer architectures already proofs this article from 2019 wrong in some ways. Even though CNNs did and do a good job, they will most likely be superseded by vision transformers. Of course you could argue that stable diffusion models and LLMs also have some domain knowledge inside. But it might be a little less with multimodal networks. So I guess the lesson should be: Don't put more domain knowledge inside than what's needed to make a product out of it right now, because everything else won't progress the field.