3 ms·
Going really big and figuring out how to go small is also working; see Lottery Tickets, ensembles, distillation, and so on.
by PartiallyTyped 3y ago
Going really big and figuring out how to go small is also working; see Lottery Tickets, ensembles, distillation, and so on.
- tysam_and 3y agoAs someone who holds a speed WR with a very tiny model (<~8.5 MB or so), I would argue kindly that Sutton's lesson is about methods that scale -- and scale goes in the opposite direction too! If you do not have solid scaling, your link between micro methods predicting mega methods is broken. Hence, Sutton's bitter law forms the foundation for a few other lemmas that I think underpin what makes really effective research (which is iteration time, and how we effectively reduce it as much as possible and make it as accessible as possible -- which thankfully for ML algorithms seems to go hand in hand! <3 :')))) )