Valid, but how much can you distill a model while retaining (or refining) usefulness? You don't need to know how to paint Rembrandt to draw an anime girl, and y

Valid, but how much can you distill a model while retaining (or refining) usefulness? You don't need to know how to paint Rembrandt to draw an anime girl, and you don't need to know about the biological taxa of North American Ducks to write a for loop. The cross-over might come to a point where you might need to explain what "duck typing" is, but even then, you only need to know that a duck is something which quacks (What does "quack" mean? Who knows..) If a model forgets how to speak French but gets much better at generating unit tests, that might be perfectly fine for the type of work we want the model to perform. The problem is we can't easily know what the model "forgets" when it gets better at doing something else. The best thing we can do is benchmark/measure their output and hope that those benchmarks cover what users care about. I suspect high quality benchmarks will quickly become almost as important as the tuning process itself.