4 ms·
> I wonder how often the opposite happens, and model designers have been able to successfully identity that "in order to do X, we need capabilities A, B and C,
by Jensson 3y ago
> I wonder how often the opposite happens, and model designers have been able to successfully identity that "in order to do X, we need capabilities A, B and C, and in order to get capabilities A, B and C, we need new datasets P & Q" ? In a hypothetical case like that they'd have been able to measure progress towards capability X.
As far as I know before transformers that just didn't happen at all since models couldn't contain that many separate skills without them interfering with each other. This whole thing with models having many high quality capabilities is still pretty new.
But yeah, I think we need something like that and all the best LLMs today probably already do it, but none have said what they do so that is just speculation.
- HarHarVeryFunny 3y agoI'm sure that there are plenty of cases where model designers are at least attempting to pursue new capabilities in a targeted approach (but how often to this degree of complexity?), while at the same time realizing that new models+datasets will also have unanticipated new capabilities. Before LLMs (whether transformer-based or not) most NNs were built to perform single tasks - a single objective, so having multiple higher-level capabilities was essentially out of the question. Of course LLMs nominally only have a single objective too, predict next word, but really they are targeting language. In the GOFAI era of rule-based symbolic AI there were also some systems/approaches that had multiple skills (e.g. expert systems like CYC, or cognitive architectures like SOAR), so maybe there are forgotten lessons there on decomposability of skills.