4 ms·
> It's not practical to manually design an ideal, on-demand, just-right limited-parameter model for every dataset we are presented with. What if that could be
by skybrian 4mo ago
> It's not practical to manually design an ideal, on-demand, just-right limited-parameter model for every dataset we are presented with.
What if that could be automated?
- rbanffy 4mo agoThe process would need to have some knowledge of the desired outcome, much like a human expert would have a hunch of the design decisions to make.
- Scene_Cast2 4mo agoNot necessarily. I'm a proponent of (admittedly not very popular) methodology of "train, do interpretability analysis, adjust model architecture". It's not more popular for a few reasons: 1) you first need to train a full general model anyhow 2) interpretability is nontrivial and not guaranteed 3) once you make the architectural changes, you can't commit to that architecture as you might miss out in the future with more advancements 4) with modern transformers, there is limited amount of architectural "play" happening.