4 ms·

It'd be fun to see where different people draw the line between linguistics and math, but I live in perpetual fear of reinventing a wheel that is very much unde

It'd be fun to see where different people draw the line between linguistics and math, but I live in perpetual fear of reinventing a wheel that is very much understood in another discipline. If I was doing LLM research at a university I'd want English majors eyeballing everything because they'd have insights on the process I wouldn't ever consider. I was in the peanut gallery for a few planning sessions on reworking documentation at a major org and kept pitching the idea of hiring library science masters. Literally what they trained for and their other job prospects pay like $35k/yr. Shot down over fears that they wouldn't "understand the code", like that's the hard part of this job.

famouswaffles 3y ago
It's not really about linguistics vs math or English majors "not understanding the code". It's that English Majors don't understand how language works. Of course, neither do ML Researchers but their methods aren't contingent on understanding the problem space so that doesn't really matter. Grammar is not how language works. It's useful fiction. English Majors have little to no "special insight" to give. When they do give "insight", it's more likely to derail the process than help it. This is the conclusion after decades of trying it the way you imagine.
- gopher_space 3y ago
  I think we're talking about two different perspectives. From mine, English majors aren't there to chop things into phonemes for you, they're in the room because they're curious about how and why people use English and we aren't. We aren't even qualified to wonder what they'd come up with. If you'd like to put the concept in a nutshell, talk to a high school teacher about your onboarding problems.
  - casercaramel144 3y ago
    http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html This has been tried for forever and never works. In fact, LLM's emerged from the *rejection* of traditional linguistics based techniques in favor of Moore's law based methods. In general, it seems that more general methods seem to win given enough data and compute (since the domain knowledge on complex problems are too difficult to encode into our programs and models). Getting English majors curious about mechanics and motivation of English is counter-productive because of their knowledge and flawed understanding of how language works. Speak for yourself, but we're very much qualified to opine on this since we have a 70 year track record of having linguists in the room never working. In the same vein, I highly doubt high school teachers have any special insights into the most effective way to implement instructing LLMs or transfer learning.
    - gopher_space 3y ago
      Looking back through this thread I should have highlighted the distinction between English and Anthropology. > Speak for yourself, but we're very much qualified to opine on this since we have a 70 year track record of having linguists in the room never working. 70 years working next to another discipline and you didn't learn anything? > In the same vein, I highly doubt high school teachers have any special insights into the most effective way to implement instructing LLMs or transfer learning. The idea is to build analogies between the two domains so you can see how they approach similar or interesting problems. You're rummaging through Education's garage, looking for tools to steal. You're the one who's supposed to have the useful insights.
      - casercaramel144 3y ago
        > 70 years working next to another discipline and you didn't learn anything? Exactly. That's the whole point of what we've been trying to say. Linguists in the room aren't just unhelpful, but counterproductive. It's actively better that their wrong ideas of human language aren't included into models. For a concrete example of this, see: https://en.wikipedia.org/wiki/Parse_tree https://en.wikipedia.org/wiki/Parse_tree People have been trying to incorporate grammar tree's into models for forever and a half. Every time it's been tried it either is actively worse than letting the model come up with it's own representation, or slightly better at best if our model isn't complex enough. >The idea is to build analogies between the two domains so you can see how they approach similar or interesting problems. You're rummaging through Education's garage, looking for tools to steal. The whole problem is that there's more garbage in education to sift through than just coming up with new ideas by rummaging through DL literature. If you check the CS section of Arxiv, there's a million times the amount of ideas that haven't been fully explored and unlike linguistics or Education, actually have a decent chance of working. The whole point is that the field is 99% garbage for computational based applications, and researcher time is much better spent doing *literally anything else*.
- tbalsam 3y ago
  Right, language quickly moves beyond grammar within a certain point in the compressive regime of a model I.M.P.E., at leasts. <3 :'))))

famouswaffles 3y ago

It's not really about linguistics vs math or English majors "not understanding the code". It's that English Majors don't understand how language works. Of course, neither do ML Researchers but their methods aren't contingent on understanding the problem space so that doesn't really matter. Grammar is not how language works. It's useful fiction. English Majors have little to no "special insight" to give. When they do give "insight", it's more likely to derail the process than help it. This is the conclusion after decades of trying it the way you imagine.

gopher_space 3y ago
I think we're talking about two different perspectives. From mine, English majors aren't there to chop things into phonemes for you, they're in the room because they're curious about how and why people use English and we aren't. We aren't even qualified to wonder what they'd come up with. If you'd like to put the concept in a nutshell, talk to a high school teacher about your onboarding problems.
- casercaramel144 3y ago
  http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html This has been tried for forever and never works. In fact, LLM's emerged from the *rejection* of traditional linguistics based techniques in favor of Moore's law based methods. In general, it seems that more general methods seem to win given enough data and compute (since the domain knowledge on complex problems are too difficult to encode into our programs and models). Getting English majors curious about mechanics and motivation of English is counter-productive because of their knowledge and flawed understanding of how language works. Speak for yourself, but we're very much qualified to opine on this since we have a 70 year track record of having linguists in the room never working. In the same vein, I highly doubt high school teachers have any special insights into the most effective way to implement instructing LLMs or transfer learning.
  - gopher_space 3y ago
    Looking back through this thread I should have highlighted the distinction between English and Anthropology. > Speak for yourself, but we're very much qualified to opine on this since we have a 70 year track record of having linguists in the room never working. 70 years working next to another discipline and you didn't learn anything? > In the same vein, I highly doubt high school teachers have any special insights into the most effective way to implement instructing LLMs or transfer learning. The idea is to build analogies between the two domains so you can see how they approach similar or interesting problems. You're rummaging through Education's garage, looking for tools to steal. You're the one who's supposed to have the useful insights.
    - casercaramel144 3y ago
      > 70 years working next to another discipline and you didn't learn anything? Exactly. That's the whole point of what we've been trying to say. Linguists in the room aren't just unhelpful, but counterproductive. It's actively better that their wrong ideas of human language aren't included into models. For a concrete example of this, see: https://en.wikipedia.org/wiki/Parse_tree https://en.wikipedia.org/wiki/Parse_tree People have been trying to incorporate grammar tree's into models for forever and a half. Every time it's been tried it either is actively worse than letting the model come up with it's own representation, or slightly better at best if our model isn't complex enough. >The idea is to build analogies between the two domains so you can see how they approach similar or interesting problems. You're rummaging through Education's garage, looking for tools to steal. The whole problem is that there's more garbage in education to sift through than just coming up with new ideas by rummaging through DL literature. If you check the CS section of Arxiv, there's a million times the amount of ideas that haven't been fully explored and unlike linguistics or Education, actually have a decent chance of working. The whole point is that the field is 99% garbage for computational based applications, and researcher time is much better spent doing *literally anything else*.
tbalsam 3y ago
Right, language quickly moves beyond grammar within a certain point in the compressive regime of a model I.M.P.E., at leasts. <3 :'))))