5 ms·

NLP researcher here. It's great to see many offerings for courses and tutorials, and NLP has made a lot of progress, in terms of both its science as well as its

by jll29 4y ago

NLP researcher here. It's great to see many offerings for courses and tutorials, and NLP has made a lot of progress, in terms of both its science as well as its re-usable software artifacts (ibraries & notebooks, standalone tools). But what saddens me is too many people are trying to dive into NLP without trying to understand language & linguistics first. For example, you can run a part of speech (POS) tagger in three lines of Python, but you will still not know much about what parts of speech are, which languages have which ones, what function they have in linguistic theory or practical applications. What are the advantages of using the C7 tagset over the C5 or PENN tagsets? Why is AT sometimes called DET? etc. I recommend people spend a bit of time to read an(y) introduction to linguistics textbook before diving into NLP, then the second investment will be worth so much more.

xtiansimon 4y ago
“I recommend people spend a bit of time to read an(y) introduction to linguistics textbook…” Linguists is a broad area of study. Can you be more specific? Such as grammar and syntax.
PainfullyNormal 4y ago
> I recommend people spend a bit of time to read an(y) introduction to linguistics textbook Do you have a favorite you can recommend?
- mothcamp 4y ago
  In addition to Jurafsky and Martin (https://web.stanford.edu/~jurafsky/slp3/ https://web.stanford.edu/~jurafsky/slp3/), I also like Emily Bender's book: https://www.goodreads.com/book/show/18128399-linguistic-fundamentals-for-natural-language-processing https://www.goodreads.com/book/show/18128399-linguistic-fund... Bender's book is NOT an end-to-end text though imo. It's more a central jumping off point. So you can read about a concept and if it sounds interesting, search more about it.
- rmellow 4y ago
  In addition to Jurafsky and Martin, I recommend Foundations of Statistical NLP by Manning and Schutze: https://nlp.stanford.edu/fsnlp/promo/ https://nlp.stanford.edu/fsnlp/promo/
- sam_lowry_ 4y ago
  Elements by Tesnière. I am not kidding, there is a shitload of knowledge there, largely forgotten by the time NLP merged with CompSci. Jurafsky and Martin, Manning and Schutze are great books for computer scientists but these do not teach about the language.
  - deleted 4y ago
    [deleted]
amitport 4y ago
NLP is a vast field nowadays, you can solve a research problem with a novel transformer architecture (for example) without knowing anything about linguistics. The thing is that NLP is a vast field and there is plenty room to go around (same goes for vision, you don't really need classical vision background as much as you used to). (also an NLP researcher. Knows nothing about linguistics)
vb234 4y ago
Could you recommend a good introduction to NLP book?
philophyse 4y ago
In your opinion, would George Yule's The Study of Language be a good introduction to linguistics? Or is there any other book that you would recommend to someone who has little knowledge of the field, but a lot of interest?
- photonemitter 4y ago
  Jumping in on this; I’ve found jurafsky/martin a good place to start. Covers a lot of ground and is a pretty good read as well. https://web.stanford.edu/~jurafsky/slp3/ https://web.stanford.edu/~jurafsky/slp3/
  - ninjin 4y ago
    As a somewhat established researcher in the field, I second Jurasky and Martin. It is peerless and what I recommend to anyone joining my team if they think their background NLP knowledge is a bit on the weak side.
mywaifuismeta 4y ago
I'm generally not a fan of these kind of high-level tutorials that tell you "use X library to get Y result" - it's just not good for learning. But any content that tries to sell you on learning ML/NLP/etc in a few weeks is just that. I understand people want to make money by targeting a large audience, but it makes me sad when I see (the vast vast majority) of practitioners not having any understanding about ML (or NLP) and just blindly applying libraries. I don't think you necessarily need a linguistics background for NLP, but I think you need either a strong linguistics OR ML background so that you know what's going on under the hood and can make connections. Anyone can call into Huggingface, you don't need a course for that.
- scarface74 4y ago
  Everything eventually gets boiled down to libraries. The purpose of technology is to get things done. I could say the same that it makes me sad that today’s developers use high level languages without ever knowing assembly. A chip designer could say that assembly language developers are saddened that the assembly language programmer never had to learn how processors are created.
  - mirker 4y ago
    It’s fine when the library is a tight abstraction. Unfortunately, ML libraries are leaky. Example: take a classification model and change the output dimensions without understanding the model.
    - nmstoker 4y ago
      Yes, the challenge people then face is that if they lack too much intuition for the subject, they can't spot obvious issues. We've all seen how ML people don't necessarily have to have the skills to solve a problem (ie i don't need to speak Vietnamese to make a "passable" ML translator) but it's not long before the lack of knowledge starts to show up embarrassing shortfalls - being too arms length about any topic is a recipe for disaster!
- Der_Einzige 4y ago
LunaSea 4y ago
Is this still true in an era where most NLP problems use language models as a solution?
- gattilorenz 4y ago
  I think so. First of all, knowing some linguistics will teach you terms and concepts (e.g. parse tree, phrase, morpheme, phoneme, etc) that will both help you find relevant literature and avoid reinventing terms for stuff that is widely known (so others will more readily find your work). Language models are currently the best solution for many problems, but it's hard to predict how we will move forward from here. Maybe the inclusion of linguistic information, or linguistic-inspired knowledge, or whatever, will be the key to having better results, or saving training time/resources. With no linguistics background, I imagine it's hard to get ideas going in that direction (and test if it's actually a good direction)
  - mothcamp 4y ago
    I agree. I think having linguistics knowledge can help especially in applied situations. Linguistics knowledge can help create fallback systems when an ML system fails, or help build rules to amplify or dampen the confidence of a response from an ML system, or aid in the engineering of a system (all that comes before or after the ML blackbox). Sort of like an algorithmic trader knowing market microstructure intimately (versus only pure statistics).
- k8si 4y ago
  Language models as a solution to what problems? Yes, you can easily use AutoModel.from_pretrained('bert-base-uncased') to convert some text into a vector of floats. What then? What are the properties of downstream (aka actually useful) datasets that might make few-shot transfer difficult or easy? How much data do your users need to provide to get a useful classifier/tagger/etc. for their problem domain? Why do seemingly-minor perturbations like typos or concating a few numbers result in major differences in representations, and how do you detect/test/mitigate this to ensure model behavior doesn't result weird downstream system behavior? How do you train a dialog system to map 'I'm good, thanks' to 'no'? How do you train a sentiment classifier learn from contextual/pragmatic cues rather than purely lexical ones (example: 'I hate to say it but this product solves all my problems.' - positive or negative sentiment?) How bad is the user experience of your Arabic-speaking customers compared to that of your English-speaking customers, and what can you do to measure this and fix it? My linguistics background really helps me think through a lot of these 'applied' NLP problems. Knowing how to make matmuls fast on GPUs and knowing exactly how multihead self-attention works is definitely useful too, but that's only one piece of building systems with NLP components.
true_religion 4y ago
I don’t know if anyone wants to dive into NLP as much as they just want to solve their problem at hand. You are right that lack of fundamental knowledge is problematic, especially that tools can allow you to make a greater quantity of solutions and therefore also a greater quantity of mistakes. However, at least the problem is still being solved. For example, a few months ago I wanted to organize my media collection by tagging files with artist names. I had a list of artist names but it wasn’t comprehensive so I wired together a bunch of python NLP libraries together to automatically pull out proper nouns from filenames, recognize English names, then annotate the files. I know almost nothing about parts of speech or anything else, so I made mistakes. About 10% of the results were errors in the first run, but after tuning it was down to about 1% which was good enough to run over the entire media library. If not for the tools, I would have never been able to finish that chore in a single day. To me, it was worth it despite my amateur mistakes. I view the library just like any other tool: a screw driver, a hammer, a wrench. I’m not a plumber or a carpenter, or an NLP researcher but I still want to use tools to fix my leaky faucets, remount my leaning cabinet doors, and organize my media collections as weekend projects.
adamsmith143 4y ago
I'm not at all sympathetic to this viewpoint. The Deep Learning revolution has shown us time and time again that Deep Learning experts universally outperform SME on modelling performance. I an almost 100% certain that the teams building big Transformers which are now by far the best NLP models (OpenAI, Meta, Google Brain, Deepmind, etc) are not made up of linguistic experts but Deep Learning experts.
- amitport 4y ago
  These groups are not mutually exclusive.
  - sam_lowry_ 4y ago
    In practice, they are, AFAIK.
  - adamsmith143 4y ago
    Maybe not but I'd guess that in this context the marginal gain for learning more about Linguistics is going to be dwarfed by learning more about Deep Learning.
meristem 4y ago
Do you have specific book suggestions?
screye 4y ago
I makes sense to completely disregard language when looking at modern NLP solutions. In some sense, 'hand engineering' anything is looked down upon. Transformers and scaling laws have made it such that the only thing that truly matters is your ability to build a model that can computationally and parametrically scale. The 2nd would be to figure out how to make more data 'viable' for usable within such a hungry model's encoding. Look at anyone who has written the last 20 seminal papers in NLP, and almost none of them have a strong background in linguistics. Vision went through a similar period of forced obsolescence, during the 2012-2016 Alexnet -> VGG -> Inception -> Resnet transition. It is unfortunate. But, time is limited and most researchers can only spare enough time to learn a few new things. Unfortunately for linguistics, it does not rank that high.
ad404b8a372f2b9 4y ago
"Every time I fire a linguist, the performance of the speech recognizer goes up" - Frederick Jelinek

by jll29 4y ago

xtiansimon 4y ago

“I recommend people spend a bit of time to read an(y) introduction to linguistics textbook…” Linguists is a broad area of study. Can you be more specific? Such as grammar and syntax.

PainfullyNormal 4y ago

> I recommend people spend a bit of time to read an(y) introduction to linguistics textbook Do you have a favorite you can recommend?

mothcamp 4y ago
In addition to Jurafsky and Martin (https://web.stanford.edu/~jurafsky/slp3/ https://web.stanford.edu/~jurafsky/slp3/), I also like Emily Bender's book: https://www.goodreads.com/book/show/18128399-linguistic-fundamentals-for-natural-language-processing https://www.goodreads.com/book/show/18128399-linguistic-fund... Bender's book is NOT an end-to-end text though imo. It's more a central jumping off point. So you can read about a concept and if it sounds interesting, search more about it.
rmellow 4y ago
In addition to Jurafsky and Martin, I recommend Foundations of Statistical NLP by Manning and Schutze: https://nlp.stanford.edu/fsnlp/promo/ https://nlp.stanford.edu/fsnlp/promo/
sam_lowry_ 4y ago
Elements by Tesnière. I am not kidding, there is a shitload of knowledge there, largely forgotten by the time NLP merged with CompSci. Jurafsky and Martin, Manning and Schutze are great books for computer scientists but these do not teach about the language.
- deleted 4y ago
  [deleted]

amitport 4y ago

NLP is a vast field nowadays, you can solve a research problem with a novel transformer architecture (for example) without knowing anything about linguistics. The thing is that NLP is a vast field and there is plenty room to go around (same goes for vision, you don't really need classical vision background as much as you used to). (also an NLP researcher. Knows nothing about linguistics)

vb234 4y ago

Could you recommend a good introduction to NLP book?

philophyse 4y ago

In your opinion, would George Yule's The Study of Language be a good introduction to linguistics? Or is there any other book that you would recommend to someone who has little knowledge of the field, but a lot of interest?

photonemitter 4y ago
Jumping in on this; I’ve found jurafsky/martin a good place to start. Covers a lot of ground and is a pretty good read as well. https://web.stanford.edu/~jurafsky/slp3/ https://web.stanford.edu/~jurafsky/slp3/
- ninjin 4y ago
  As a somewhat established researcher in the field, I second Jurasky and Martin. It is peerless and what I recommend to anyone joining my team if they think their background NLP knowledge is a bit on the weak side.

mywaifuismeta 4y ago

I'm generally not a fan of these kind of high-level tutorials that tell you "use X library to get Y result" - it's just not good for learning. But any content that tries to sell you on learning ML/NLP/etc in a few weeks is just that. I understand people want to make money by targeting a large audience, but it makes me sad when I see (the vast vast majority) of practitioners not having any understanding about ML (or NLP) and just blindly applying libraries. I don't think you necessarily need a linguistics background for NLP, but I think you need either a strong linguistics OR ML background so that you know what's going on under the hood and can make connections. Anyone can call into Huggingface, you don't need a course for that.

scarface74 4y ago
Everything eventually gets boiled down to libraries. The purpose of technology is to get things done. I could say the same that it makes me sad that today’s developers use high level languages without ever knowing assembly. A chip designer could say that assembly language developers are saddened that the assembly language programmer never had to learn how processors are created.
- mirker 4y ago
  It’s fine when the library is a tight abstraction. Unfortunately, ML libraries are leaky. Example: take a classification model and change the output dimensions without understanding the model.
  - nmstoker 4y ago
    Yes, the challenge people then face is that if they lack too much intuition for the subject, they can't spot obvious issues. We've all seen how ML people don't necessarily have to have the skills to solve a problem (ie i don't need to speak Vietnamese to make a "passable" ML translator) but it's not long before the lack of knowledge starts to show up embarrassing shortfalls - being too arms length about any topic is a recipe for disaster!
Der_Einzige 4y ago

LunaSea 4y ago

Is this still true in an era where most NLP problems use language models as a solution?

gattilorenz 4y ago
I think so. First of all, knowing some linguistics will teach you terms and concepts (e.g. parse tree, phrase, morpheme, phoneme, etc) that will both help you find relevant literature and avoid reinventing terms for stuff that is widely known (so others will more readily find your work). Language models are currently the best solution for many problems, but it's hard to predict how we will move forward from here. Maybe the inclusion of linguistic information, or linguistic-inspired knowledge, or whatever, will be the key to having better results, or saving training time/resources. With no linguistics background, I imagine it's hard to get ideas going in that direction (and test if it's actually a good direction)
- mothcamp 4y ago
  I agree. I think having linguistics knowledge can help especially in applied situations. Linguistics knowledge can help create fallback systems when an ML system fails, or help build rules to amplify or dampen the confidence of a response from an ML system, or aid in the engineering of a system (all that comes before or after the ML blackbox). Sort of like an algorithmic trader knowing market microstructure intimately (versus only pure statistics).
k8si 4y ago
Language models as a solution to what problems? Yes, you can easily use AutoModel.from_pretrained('bert-base-uncased') to convert some text into a vector of floats. What then? What are the properties of downstream (aka actually useful) datasets that might make few-shot transfer difficult or easy? How much data do your users need to provide to get a useful classifier/tagger/etc. for their problem domain? Why do seemingly-minor perturbations like typos or concating a few numbers result in major differences in representations, and how do you detect/test/mitigate this to ensure model behavior doesn't result weird downstream system behavior? How do you train a dialog system to map 'I'm good, thanks' to 'no'? How do you train a sentiment classifier learn from contextual/pragmatic cues rather than purely lexical ones (example: 'I hate to say it but this product solves all my problems.' - positive or negative sentiment?) How bad is the user experience of your Arabic-speaking customers compared to that of your English-speaking customers, and what can you do to measure this and fix it? My linguistics background really helps me think through a lot of these 'applied' NLP problems. Knowing how to make matmuls fast on GPUs and knowing exactly how multihead self-attention works is definitely useful too, but that's only one piece of building systems with NLP components.

true_religion 4y ago

I don’t know if anyone wants to dive into NLP as much as they just want to solve their problem at hand. You are right that lack of fundamental knowledge is problematic, especially that tools can allow you to make a greater quantity of solutions and therefore also a greater quantity of mistakes. However, at least the problem is still being solved. For example, a few months ago I wanted to organize my media collection by tagging files with artist names. I had a list of artist names but it wasn’t comprehensive so I wired together a bunch of python NLP libraries together to automatically pull out proper nouns from filenames, recognize English names, then annotate the files. I know almost nothing about parts of speech or anything else, so I made mistakes. About 10% of the results were errors in the first run, but after tuning it was down to about 1% which was good enough to run over the entire media library. If not for the tools, I would have never been able to finish that chore in a single day. To me, it was worth it despite my amateur mistakes. I view the library just like any other tool: a screw driver, a hammer, a wrench. I’m not a plumber or a carpenter, or an NLP researcher but I still want to use tools to fix my leaky faucets, remount my leaning cabinet doors, and organize my media collections as weekend projects.

adamsmith143 4y ago

I'm not at all sympathetic to this viewpoint. The Deep Learning revolution has shown us time and time again that Deep Learning experts universally outperform SME on modelling performance. I an almost 100% certain that the teams building big Transformers which are now by far the best NLP models (OpenAI, Meta, Google Brain, Deepmind, etc) are not made up of linguistic experts but Deep Learning experts.

amitport 4y ago
These groups are not mutually exclusive.
- sam_lowry_ 4y ago
  In practice, they are, AFAIK.
- adamsmith143 4y ago
  Maybe not but I'd guess that in this context the marginal gain for learning more about Linguistics is going to be dwarfed by learning more about Deep Learning.

meristem 4y ago

Do you have specific book suggestions?

screye 4y ago

I makes sense to completely disregard language when looking at modern NLP solutions. In some sense, 'hand engineering' anything is looked down upon. Transformers and scaling laws have made it such that the only thing that truly matters is your ability to build a model that can computationally and parametrically scale. The 2nd would be to figure out how to make more data 'viable' for usable within such a hungry model's encoding. Look at anyone who has written the last 20 seminal papers in NLP, and almost none of them have a strong background in linguistics. Vision went through a similar period of forced obsolescence, during the 2012-2016 Alexnet -> VGG -> Inception -> Resnet transition. It is unfortunate. But, time is limited and most researchers can only spare enough time to learn a few new things. Unfortunately for linguistics, it does not rank that high.

ad404b8a372f2b9 4y ago

"Every time I fire a linguist, the performance of the speech recognizer goes up" - Frederick Jelinek