4 ms·
No, not really, there was a lot of engineering work and bunch of not-so-big ideas (e.g. InstructGPT reinforcement learning after the model's training), but you
by PeterisP 4y ago
No, not really, there was a lot of engineering work and bunch of not-so-big ideas (e.g. InstructGPT reinforcement learning after the model's training), but you can go from the transformers paper to current state of art without needing a "big idea".
And I think this is the major "big idea", accepting the bitter lesson (http://incompleteideas.net/IncIdeas/BitterLesson.html http://incompleteideas.net/IncIdeas/BitterLesson.html) that major user-visible progress and new emerging capabilities doesn't necessarily require any big ideas but simply scaling to more compute.
- mdp2021 4y agoI disagree. And first of all, there is a reflective meta teaching from the very idea of the "Bitter Lessons": the past reveals that (in a way) "the application of models has not been a winner" - but we cannot really know that it is not, because we do not have obtained a model out of it, a model that shows why, an explanation - epistemologically, the "discouraging" protocols cannot be made a "law". Practically, there still is a need to identify the proper architecture(s) to avoid the undesired weaknesses of the attempts in the current stages.