15 ms·

How to solve most NLP problems: a step-by-step guide

461 points by e_ameisen 9y ago

odonnellryan 9y ago
I am not sure how many people have an issue with this, but it seems to me that computer science, just over the relatively short time I've been paying attention, is becoming more-and-more abstract in a lot of ways. You can code something incredibly complex that works great without understanding any of the math underneath. Understanding the math arguably makes you a better engineer overall, but isn't required to solve many of these problems. I think it's pretty cool, but I'm sure a lot of people have a big issue with the "just TRUST the library!" approach.
- testouts 9y ago
  yes , in the new world we will need shitty engineers and good engineers ... and we will not need ditch diggers as cashiers as much but we ill make the shitty engineering jobs as easy as being a cashier. and life goes on...
- hathawsh 9y ago
  Surely this is a good thing, though. Computers now span every field and it's not feasible to know the math behind all types of software. It's better to go deep in a few areas and let other people take care of the rest.
- zzz95 9y ago
  I understand what you are saying and I agree with it too. What you are worried about, is what happens in the short term. But, in the long term, as the technology matures, this is what should be happening. It will become an easy to use black-box which works for almost 99.99% of the cases and gracefully handles the remaining cases. You will no longer need PhDs to use the black-box, only to develop them. For e.g., look at numerical analyses toolboxes, like automatic integrator. Even if you do not understand the underlying system of differential equations, the libraries just work or throw a nice legible error. They encapsulate a ton of ugliness inside, but that is all right as long as the solutions are predictable.
  - has2k1 9y ago
    Really interesting view point. When the convenience of use improves along with the performance of the methods, then we worry less about not understanding how the methods work. In the broad sense, isn't this an example of how we trust?
- IanCal 9y ago
  This has been the case throughout the history of computers and many other things. We build layers on top of layers on top of layers. Occasionally this causes problems. Often it solves them.
- TeMPOraL 9y ago
  Layers are good. They allow our limited mind to deal with more and more complex problems. The issue forms when layers are inadequately described, or blackbox things they shouldn't. The worst form of this issue is when people start writing introductory documentation as if it was a marketing copy. A fair introduction to a library would be like this: "This library lets you make X, Y and Z from A, B and C. It does so using mathematical methods 1, 2, 3 - therefore, it will be great for this-and-that type of A-B-C, but will not perform well for some-other-type." Such a description will tell you where the limits of applicability are, and what to look for if you want to understand more. (Also, neural network libs should come with a big, bold caveat: "this is magic, only 10 people on the planet know how the whole stack works; the rest of us just perform a ritual on a cluster of GPUs and pray that thus summoned entity will do sorta ok-ish job with our problem (and if it doesn't, you're on your own)".)
- amelius 9y ago
  > it seems to me that computer science, just over the relatively short time I've been paying attention, is becoming more-and-more abstract in a lot of ways. But can we still call it computer science? It seems to me more like linguistics / mathematics / statistics, with the CS only to do the low-level computations and accounting of data. Of course, if you build a global-scale efficient search engine with it, the role of CS may become bigger.
  - maksimum 9y ago
    As a computer scientist I think we benefit as a group from having NLP fit under computer science, so yes I think we should call it computer science. The overarching question is: why does it matter what we classify NLP as?
    - amelius 9y ago
      I think it matters because it determines what kind of people you would hire to do the R&D.
    - chaoky 9y ago
      NLP is a domain specific problem. Of course it shouldn't be under computer science. Pure computer scientists are much less useful than linguists for these things. That's like arguing that building physics simulations is under the realm of computer science. You'd want your team to be mostly linguists, some which specialize in computational linguistics. A vanilla computer scientist, to be frank, is almost useless, especially at the PhD level.
- losteric 9y ago
  The key is building the right abstraction layers - clean separation of responsibilities into black boxes. Outside of security, no one complains about trusting the heap of abstractions that interpreted languages depend on... interpreters, compilers, operating systems, hardware... clear layers. I don't think machine learning and NLP has reached that point. Largely because the field is based on probabilities and tunable parameters which are hard to expose without complexity, and hard to trust when they aren't exposed.
- fnl 9y ago
  Well, the 80% accuracy the author reaches isn't anywhere near what you need in production for text classification. Imagine classifying all of Twitter with that... I don't want to make this look bad - the author did a great job at this "introduction to text mining" article. But the results are a far cry form prediction ready, as I hope would be obvious: Anything but a tiny false positive rate will just produce infinite false alarms (or miss all relevant cases).
Rickasaurus 9y ago
Bag of words is the death of comprehensible NLP
- jventura 9y ago
  Agree.. There's also some things that you can't do with "bag-of-words" methods, such as handling multi-words because for those, the order of words is relevant.
  - samfriedman 9y ago
    bag of n-grams can help here, but yes you will still run into a wall quick if you require phrase understanding or context.
    - aglionby 9y ago
      That has its own set of problems, namely data sparsity and resulting in high dimensionality vectors, although like you say does give you a little bit of context. I'm not sure that bag of anything is a good solution.
paulsutter 9y ago
NLP is one of the most challenging areas of research, and nothing in this article will help solve even 0.009% of those challenges Example of the wisdom herein: > Remove words that are not relevant, such as “@” twitter mentions or urls
- agentgt 9y ago
  Yes that title isn't just pedantically incorrect... its extremely disingenuous. Where did the 90% even come from?
- tanilama 9y ago
  My thought is exactly. While the content is not bad, the title is a clickbait. It is just how people deal with text classification.
- e_ameisen 9y ago
  Hi, Author here, happy to answer questions and share our vision of it. Many problems require more complex approaches, and we definitely have Fellows tackle some of those (https://blog.insightdatascience.com/entity2vec-dad368c5b830 https://blog.insightdatascience.com/entity2vec-dad368c5b830). That being said, when it comes to the volume of practical applications that come up for the many teams that we work with, the vast majority can be solved by the techniques outlined in the post. These techniques are simpler, but often overlooked. We believe that they should often be a starting point, and most of the time they end up being good enough for the job.
  - deleted 9y ago
    [deleted]
  - paulsutter 9y ago
    A better title for your article might be “How to solve most text-processing problems”, because you’re really not talking about NLP.
    - fnl 9y ago
minimaxir 9y ago
Word2Vec and bag-of-words/tf-idf are somewhat obsolete in 2018 for modeling. For classification tasks, fasttext (https://github.com/facebookresearch/fastText https://github.com/facebookresearch/fastText) performs better and faster. Fasttext is also available in the popular NLP Python library gensim, with a good demo notebook: https://radimrehurek.com/gensim/models/fasttext.html https://radimrehurek.com/gensim/models/fasttext.html And of course, if you have a GPU, recurrent neural networks (or other deep learning architectures) are the endgame for the remaining 10% of problems (a good example is SpaCy's DL implementation: https://spacy.io/ https://spacy.io/). Or use those libraries to incorporate fasttext for text encoding, which has worked well in my use cases.
- taneq 9y ago
  I'm not an expert but I'd be surprised if you could declare any approach "endgame" at this point.
  - sundarurfriend 9y ago
    My interpretation is that they meant something like "last resort" there.
- tanilama 9y ago
  Fasttext is not that different from word2vec
  - crossroads091 9y ago
    Fasttext's ability to handle OOVs, however naïve, cannot be understated. I'm not sure about the literature on this, but I think generalizing based on character-gram wordvectors is probably better than assigning a dropout based common OOV vector.
    - tanilama 9y ago
      Evolution for sure, heck, they even has the same author. But fasttext is the same idea as word2vec, the objective is identical.
code4tee 9y ago
Good intro, but the approaches used here are quite basic and outdated for 2018. Not sure this solves 90% of NLP problems.
- minimaxir 9y ago
  Granted, even if the approaches are outdated, they can still be used to solve NLP problems.
- zootam 9y ago
  Any suggestions for good resources on more advanced and current approaches?
mikevm 9y ago
A question to the NLP experts out there -- is it possible to automatically detect various pre-defined attributes about a person by automatically analyzing relevant texts? For example, finding out whether a person is anti-capitalist by scanning his blog posts related to economics. I'm not even sure how to approach such a problem.
- e_ameisen 9y ago
  Author here. If you have the means to label some data, this becomes quite an easy classification problem. Get multiple examples of posts, and labels (capitalist, libertarian, ...) for a group of users that represents your total population well, and train a classifier.
  - mikevm 9y ago
    thank you!
- bomb199 9y ago
  What you're describing is just a classifier, and you can certainly train something to do this. However, I'm quite skeptical it would work well. I suppose if you were Google and had all their data, perhaps. Putting different data types together (like text, pictures, locations) adds a lot of difficulty.
CGamesPlay 9y ago
This was an interesting read, but when I read sentences like "The words it picked up look much more relevant!", I'm reminded of the XKCD explanation of machine learning: https://xkcd.com/1838/ https://xkcd.com/1838/
phijFTW 9y ago
I think def sanitize_characters(raw, clean): for line in input_file: out = line output_file.write(line) sanitize_characters(input_file, output_file) should be def sanitize_characters(raw, clean): for line in raw: out = line clean.write(line) sanitize_characters(input_file, output_file) in your notebook: https://github.com/hundredblocks/concrete_NLP_tutorial/blob/master/NLP_notebook.ipynb https://github.com/hundredblocks/concrete_NLP_tutorial/blob/... Or am I mistaken?
hinkley 9y ago
I know natural language processing predates Neuro-linguistic programming, but I still can’t see ‘NLP’ without the little hairs on the back of my neck standing up.
fnl 9y ago
Bad title (this is all about text classification/mining, not NLP), but a very nice introduction at that. Maybe a tad optimistic - I'd never even consider applying a classifier with 80% accuracy to the Twitter firehose (unless extremely noisy performance were a non-issue - but it never is ... :-)).
paultopia 9y ago
I think this might be the first blog post I've read to actually explain how to use word vectors as features---good for the author!
master_yoda_1 9y ago
The title os the blog is to ambitious.
polm23 9y ago
The thing that jumped out at me in this article was the use of Lime to explain models - I hadn't heard of it before. https://github.com/marcotcr/lime https://github.com/marcotcr/lime For NLP tasks, it looks like what it does is selectively delete words from the input and check the classifier output. This way it determines which words have the biggest effect on the output without needing to know anything about how your model works.

461 points by e_ameisen 9y ago

I am not sure how many people have an issue with this, but it seems to me that computer science, just over the relatively short time I've been paying attention, is becoming more-and-more abstract in a lot of ways. You can code something incredibly complex that works great without understanding any of the math underneath. Understanding the math arguably makes you a better engineer overall, but isn't required to solve many of these problems. I think it's pretty cool, but I'm sure a lot of people have a big issue with the "just TRUST the library!" approach.

testouts 9y ago
yes , in the new world we will need shitty engineers and good engineers ... and we will not need ditch diggers as cashiers as much but we ill make the shitty engineering jobs as easy as being a cashier. and life goes on...
hathawsh 9y ago
Surely this is a good thing, though. Computers now span every field and it's not feasible to know the math behind all types of software. It's better to go deep in a few areas and let other people take care of the rest.
zzz95 9y ago
I understand what you are saying and I agree with it too. What you are worried about, is what happens in the short term. But, in the long term, as the technology matures, this is what should be happening. It will become an easy to use black-box which works for almost 99.99% of the cases and gracefully handles the remaining cases. You will no longer need PhDs to use the black-box, only to develop them. For e.g., look at numerical analyses toolboxes, like automatic integrator. Even if you do not understand the underlying system of differential equations, the libraries just work or throw a nice legible error. They encapsulate a ton of ugliness inside, but that is all right as long as the solutions are predictable.
- has2k1 9y ago
  Really interesting view point. When the convenience of use improves along with the performance of the methods, then we worry less about not understanding how the methods work. In the broad sense, isn't this an example of how we trust?
IanCal 9y ago
This has been the case throughout the history of computers and many other things. We build layers on top of layers on top of layers. Occasionally this causes problems. Often it solves them.
TeMPOraL 9y ago
Layers are good. They allow our limited mind to deal with more and more complex problems. The issue forms when layers are inadequately described, or blackbox things they shouldn't. The worst form of this issue is when people start writing introductory documentation as if it was a marketing copy. A fair introduction to a library would be like this: "This library lets you make X, Y and Z from A, B and C. It does so using mathematical methods 1, 2, 3 - therefore, it will be great for this-and-that type of A-B-C, but will not perform well for some-other-type." Such a description will tell you where the limits of applicability are, and what to look for if you want to understand more. (Also, neural network libs should come with a big, bold caveat: "this is magic, only 10 people on the planet know how the whole stack works; the rest of us just perform a ritual on a cluster of GPUs and pray that thus summoned entity will do sorta ok-ish job with our problem (and if it doesn't, you're on your own)".)
amelius 9y ago
> it seems to me that computer science, just over the relatively short time I've been paying attention, is becoming more-and-more abstract in a lot of ways. But can we still call it computer science? It seems to me more like linguistics / mathematics / statistics, with the CS only to do the low-level computations and accounting of data. Of course, if you build a global-scale efficient search engine with it, the role of CS may become bigger.
- maksimum 9y ago
  As a computer scientist I think we benefit as a group from having NLP fit under computer science, so yes I think we should call it computer science. The overarching question is: why does it matter what we classify NLP as?
  - amelius 9y ago
    I think it matters because it determines what kind of people you would hire to do the R&D.
  - chaoky 9y ago
    NLP is a domain specific problem. Of course it shouldn't be under computer science. Pure computer scientists are much less useful than linguists for these things. That's like arguing that building physics simulations is under the realm of computer science. You'd want your team to be mostly linguists, some which specialize in computational linguistics. A vanilla computer scientist, to be frank, is almost useless, especially at the PhD level.
losteric 9y ago
The key is building the right abstraction layers - clean separation of responsibilities into black boxes. Outside of security, no one complains about trusting the heap of abstractions that interpreted languages depend on... interpreters, compilers, operating systems, hardware... clear layers. I don't think machine learning and NLP has reached that point. Largely because the field is based on probabilities and tunable parameters which are hard to expose without complexity, and hard to trust when they aren't exposed.
fnl 9y ago
Well, the 80% accuracy the author reaches isn't anywhere near what you need in production for text classification. Imagine classifying all of Twitter with that... I don't want to make this look bad - the author did a great job at this "introduction to text mining" article. But the results are a far cry form prediction ready, as I hope would be obvious: Anything but a tiny false positive rate will just produce infinite false alarms (or miss all relevant cases).

Rickasaurus 9y ago

Bag of words is the death of comprehensible NLP

jventura 9y ago
Agree.. There's also some things that you can't do with "bag-of-words" methods, such as handling multi-words because for those, the order of words is relevant.
- samfriedman 9y ago
  bag of n-grams can help here, but yes you will still run into a wall quick if you require phrase understanding or context.
  - aglionby 9y ago
    That has its own set of problems, namely data sparsity and resulting in high dimensionality vectors, although like you say does give you a little bit of context. I'm not sure that bag of anything is a good solution.

paulsutter 9y ago

NLP is one of the most challenging areas of research, and nothing in this article will help solve even 0.009% of those challenges Example of the wisdom herein: > Remove words that are not relevant, such as “@” twitter mentions or urls

agentgt 9y ago
Yes that title isn't just pedantically incorrect... its extremely disingenuous. Where did the 90% even come from?
tanilama 9y ago
My thought is exactly. While the content is not bad, the title is a clickbait. It is just how people deal with text classification.
e_ameisen 9y ago
Hi, Author here, happy to answer questions and share our vision of it. Many problems require more complex approaches, and we definitely have Fellows tackle some of those (https://blog.insightdatascience.com/entity2vec-dad368c5b830 https://blog.insightdatascience.com/entity2vec-dad368c5b830). That being said, when it comes to the volume of practical applications that come up for the many teams that we work with, the vast majority can be solved by the techniques outlined in the post. These techniques are simpler, but often overlooked. We believe that they should often be a starting point, and most of the time they end up being good enough for the job.
- deleted 9y ago
  [deleted]
- paulsutter 9y ago
  A better title for your article might be “How to solve most text-processing problems”, because you’re really not talking about NLP.
  - fnl 9y ago

minimaxir 9y ago

Word2Vec and bag-of-words/tf-idf are somewhat obsolete in 2018 for modeling. For classification tasks, fasttext (https://github.com/facebookresearch/fastText https://github.com/facebookresearch/fastText) performs better and faster. Fasttext is also available in the popular NLP Python library gensim, with a good demo notebook: https://radimrehurek.com/gensim/models/fasttext.html https://radimrehurek.com/gensim/models/fasttext.html And of course, if you have a GPU, recurrent neural networks (or other deep learning architectures) are the endgame for the remaining 10% of problems (a good example is SpaCy's DL implementation: https://spacy.io/ https://spacy.io/). Or use those libraries to incorporate fasttext for text encoding, which has worked well in my use cases.

taneq 9y ago
I'm not an expert but I'd be surprised if you could declare any approach "endgame" at this point.
- sundarurfriend 9y ago
  My interpretation is that they meant something like "last resort" there.
tanilama 9y ago
Fasttext is not that different from word2vec
- crossroads091 9y ago
  Fasttext's ability to handle OOVs, however naïve, cannot be understated. I'm not sure about the literature on this, but I think generalizing based on character-gram wordvectors is probably better than assigning a dropout based common OOV vector.
  - tanilama 9y ago
    Evolution for sure, heck, they even has the same author. But fasttext is the same idea as word2vec, the objective is identical.

code4tee 9y ago

Good intro, but the approaches used here are quite basic and outdated for 2018. Not sure this solves 90% of NLP problems.

minimaxir 9y ago
Granted, even if the approaches are outdated, they can still be used to solve NLP problems.
zootam 9y ago
Any suggestions for good resources on more advanced and current approaches?

mikevm 9y ago

A question to the NLP experts out there -- is it possible to automatically detect various pre-defined attributes about a person by automatically analyzing relevant texts? For example, finding out whether a person is anti-capitalist by scanning his blog posts related to economics. I'm not even sure how to approach such a problem.

e_ameisen 9y ago
Author here. If you have the means to label some data, this becomes quite an easy classification problem. Get multiple examples of posts, and labels (capitalist, libertarian, ...) for a group of users that represents your total population well, and train a classifier.
- mikevm 9y ago
  thank you!
bomb199 9y ago
What you're describing is just a classifier, and you can certainly train something to do this. However, I'm quite skeptical it would work well. I suppose if you were Google and had all their data, perhaps. Putting different data types together (like text, pictures, locations) adds a lot of difficulty.

CGamesPlay 9y ago

This was an interesting read, but when I read sentences like "The words it picked up look much more relevant!", I'm reminded of the XKCD explanation of machine learning: https://xkcd.com/1838/ https://xkcd.com/1838/

phijFTW 9y ago

I think def sanitize_characters(raw, clean): for line in input_file: out = line output_file.write(line) sanitize_characters(input_file, output_file) should be def sanitize_characters(raw, clean): for line in raw: out = line clean.write(line) sanitize_characters(input_file, output_file) in your notebook: https://github.com/hundredblocks/concrete_NLP_tutorial/blob/master/NLP_notebook.ipynb https://github.com/hundredblocks/concrete_NLP_tutorial/blob/... Or am I mistaken?

hinkley 9y ago

I know natural language processing predates Neuro-linguistic programming, but I still can’t see ‘NLP’ without the little hairs on the back of my neck standing up.

fnl 9y ago

Bad title (this is all about text classification/mining, not NLP), but a very nice introduction at that. Maybe a tad optimistic - I'd never even consider applying a classifier with 80% accuracy to the Twitter firehose (unless extremely noisy performance were a non-issue - but it never is ... :-)).

paultopia 9y ago

I think this might be the first blog post I've read to actually explain how to use word vectors as features---good for the author!

master_yoda_1 9y ago

The title os the blog is to ambitious.

polm23 9y ago

The thing that jumped out at me in this article was the use of Lime to explain models - I hadn't heard of it before. https://github.com/marcotcr/lime https://github.com/marcotcr/lime For NLP tasks, it looks like what it does is selectively delete words from the input and check the classifier output. This way it determines which words have the biggest effect on the output without needing to know anything about how your model works.