3 ms·

http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html This has been tried for forever and never w

http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html This has been tried for forever and never works. In fact, LLM's emerged from the *rejection* of traditional linguistics based techniques in favor of Moore's law based methods. In general, it seems that more general methods seem to win given enough data and compute (since the domain knowledge on complex problems are too difficult to encode into our programs and models). Getting English majors curious about mechanics and motivation of English is counter-productive because of their knowledge and flawed understanding of how language works. Speak for yourself, but we're very much qualified to opine on this since we have a 70 year track record of having linguists in the room never working. In the same vein, I highly doubt high school teachers have any special insights into the most effective way to implement instructing LLMs or transfer learning.

gopher_space 3y ago
Looking back through this thread I should have highlighted the distinction between English and Anthropology. > Speak for yourself, but we're very much qualified to opine on this since we have a 70 year track record of having linguists in the room never working. 70 years working next to another discipline and you didn't learn anything? > In the same vein, I highly doubt high school teachers have any special insights into the most effective way to implement instructing LLMs or transfer learning. The idea is to build analogies between the two domains so you can see how they approach similar or interesting problems. You're rummaging through Education's garage, looking for tools to steal. You're the one who's supposed to have the useful insights.
- casercaramel144 3y ago
  > 70 years working next to another discipline and you didn't learn anything? Exactly. That's the whole point of what we've been trying to say. Linguists in the room aren't just unhelpful, but counterproductive. It's actively better that their wrong ideas of human language aren't included into models. For a concrete example of this, see: https://en.wikipedia.org/wiki/Parse_tree https://en.wikipedia.org/wiki/Parse_tree People have been trying to incorporate grammar tree's into models for forever and a half. Every time it's been tried it either is actively worse than letting the model come up with it's own representation, or slightly better at best if our model isn't complex enough. >The idea is to build analogies between the two domains so you can see how they approach similar or interesting problems. You're rummaging through Education's garage, looking for tools to steal. The whole problem is that there's more garbage in education to sift through than just coming up with new ideas by rummaging through DL literature. If you check the CS section of Arxiv, there's a million times the amount of ideas that haven't been fully explored and unlike linguistics or Education, actually have a decent chance of working. The whole point is that the field is 99% garbage for computational based applications, and researcher time is much better spent doing *literally anything else*.

gopher_space 3y ago

Looking back through this thread I should have highlighted the distinction between English and Anthropology. > Speak for yourself, but we're very much qualified to opine on this since we have a 70 year track record of having linguists in the room never working. 70 years working next to another discipline and you didn't learn anything? > In the same vein, I highly doubt high school teachers have any special insights into the most effective way to implement instructing LLMs or transfer learning. The idea is to build analogies between the two domains so you can see how they approach similar or interesting problems. You're rummaging through Education's garage, looking for tools to steal. You're the one who's supposed to have the useful insights.

casercaramel144 3y ago
> 70 years working next to another discipline and you didn't learn anything? Exactly. That's the whole point of what we've been trying to say. Linguists in the room aren't just unhelpful, but counterproductive. It's actively better that their wrong ideas of human language aren't included into models. For a concrete example of this, see: https://en.wikipedia.org/wiki/Parse_tree https://en.wikipedia.org/wiki/Parse_tree People have been trying to incorporate grammar tree's into models for forever and a half. Every time it's been tried it either is actively worse than letting the model come up with it's own representation, or slightly better at best if our model isn't complex enough. >The idea is to build analogies between the two domains so you can see how they approach similar or interesting problems. You're rummaging through Education's garage, looking for tools to steal. The whole problem is that there's more garbage in education to sift through than just coming up with new ideas by rummaging through DL literature. If you check the CS section of Arxiv, there's a million times the amount of ideas that haven't been fully explored and unlike linguistics or Education, actually have a decent chance of working. The whole point is that the field is 99% garbage for computational based applications, and researcher time is much better spent doing *literally anything else*.