6 ms·

Very relatable post. Isn't NLTK primarily a teaching / demonstration tool though? I just checked their website and the claim of "NLTK is a leading platform for

by Radim 11y ago

Very relatable post. Isn't NLTK primarily a teaching / demonstration tool though? I just checked their website and the claim of "NLTK is a leading platform for building Python programs to work with human language data... a suite of libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning" does sound a little odd. But I think everyone in the industry knows NLTK's place and purpose -- you practically cannot avoid finding out quickly. NLTK's scope is clearly too broad to be meaningfully cutting edge at any one thing. New libraries and implementations will always have an advantage. It's easier to tout "simplicity and leanness" when you don't have to carry over all the baggage and backward compatibility accumulated over the years. For that reason, an occasional "complexity reset" is expected, and if a library would not or can not do it, another library will. Will SpaCy's fate be different, 10 years down the road?

nkozyra 11y ago
NLTK is expansive, which seems to be the real, primary complaint. But it's also not $5,000 a year. I applaud the effort, and the product looks very good, but I don't think there was enough here to say, definitively, "don't use NLTK."
- TD-Linux 11y ago
  If you missed it, SpaCy is AGPLv3 as well. $5,000 a year doesn't seem so bad if you are doing something that can't work with the AGPLv3.
  - nkozyra 11y ago
    No, I saw that ... my point is that there's a profit motive that would make utilization of NLTK a non starter regardless of either platform's merits.
  - eridius 11y ago
    There's a pretty large gap between "$5,000 a year is reasonable" and "AGPLv3 is usable" (or any GPL variant). Speaking personally, the overwhelming majority of projects I work on can't ever go near GPL, because the project itself does not want to catch the awful GPL virus. Even open-source projects, nearly all of the ones I touch are not GPL but are some other, much more sane, license like Apache, BSD, or MIT.
    - belorn 11y ago
      No one here is interested in reading inflammatory posts about tiresome old flame wars. You call it an awful virus, I call proprietary licenses a distasteful cancer. You call it insane, I call it entitled attitude to other peoples work. At work and private life I don't include other peoples copyrighted work if I don't have a license for it. It's one simple rule. If I don't want to agree to the license because its too expensive or it put requirements on me which I refuse (like NDA's), then I have a simple choose to make. I can chose not use it, I can implement it myself, or I can hire someone else to do it.
      - eridius 11y ago
        You're arguing the wrong point. I'm not advocating for ignoring licenses, and I have no idea why you think I would. My point is entirely that the OP's claim that $5,000 a year if you can't use the AGPL is absolutely fucking stupid. There's tons of reasons why you can't use the AGPL and can't afford $5,000 a year. So go take your sanctimony elsewhere.
        belorn 11y ago
        If you don't want to pay the price and refuse to accept the conditions in which it is given to you for free, then tough luck!. Don't come to me crying about how you want it for free while adding your own license on top because your work is special.
        eridius 11y ago
        You really suck at reading comprehension. Please stop with this nonsense, it has no bearing whatsoever to any of my comments.
        belorn 11y ago
        You seem very angry at people who offer you free code to use. Please don't post inflammatory comments in HN in that state.
aloedrip 11y ago
Could not have said this better. New developers should not discount the benefits of hindsight, and humility goes a long way in the open source community.
danieldk 11y ago
It's easier to tout "simplicity and leanness" when you don't have to carry over all the baggage and backward compatibility accumulated over the years. Well, why do we have to build large, clunky (NLP) libraries to start with? Build lean and mean components as UNIX programs or easily bindable libraries and use a reasonable input/output format. E.g. nearly every statistical dependency parser uses CoNLL-X for input/output. You'll have no trouble swapping out MaltParser, Turbo Parser, or my neural net dependency parser. They all use the same, boring, tabular format. Sure, this could be more work for a beginner. So, a project could make a curated list / meta-package of components that are robust, state-of-the-art and work together.
- syllogism 11y ago
  I definitely agree that the NLP library shouldn't have an opinion about your machine learning, shouldn't give you every accuracy measure under the sun, etc. My original vision for spaCy was smaller. But there are two problems. 1) If I ship you a statistical model, and it's late in a pipeline, like a parser, the earlier components in the pipeline are not swappable. If you change the tokenization, POS tagging, lemmatization etc, the parser model will give you worse output. This isn't obvious to people, and the problem can be subtle. For instance, some NER models use POS tag features, others don't. 2) The output format isn't actually that convenient. It sucks that everyone has to write this tree processing code, and then aligning the tokenized output back to the original string is a pain, if you want to calculate mark-up.
- Radim 11y ago
  We're in violent agreement. It makes me sad to see neat, focused libraries have their mission blurred, API expanded, code base obfuscated... Until they satisfy everybody's use case, which is to say, they're useless. Some features are best left to user-land. It's a non-trivial tradeoff, obviously: "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can." (Zawinski's law) I actually wrote a rant on this topic a while ago: http://rare-technologies.com/data-streaming-in-python-generators-iterators-iterables/ http://rare-technologies.com/data-streaming-in-python-genera...
Smerity 11y ago
As you state from looking at the website, the stated purpose is for education, but NLTK is used (and advertised) as far more than that. Due to this mixed message, NLTK is used in places it shouldn't be - the author's main concern. For me, NLTK also has issues in education. Teaching requires clarity - a complex codebase rarely allows for that. The author wrote an article "Parsing English in 500 lines of Python"[1] which does a great job of explaining, by being simple and lean, how to parse. Additionally, it achieved the same level of accuracy as the Stanford NLP parser - a larger and more complex parser. That to me is the pinnacle of an educational objective - clear, concise, and practical. [1]: http://spacy.io/blog/parsing-english-in-python/ http://spacy.io/blog/parsing-english-in-python/
syllogism 11y ago
> Will SpaCy's fate be different, 10 years down the road? Yes --- because I consolidate my algorithms and delete dead code. I've probably written five or six times as much code as currently lives in spaCy. I hope by then spaCy will be smaller, not bigger, as we reach a more concise understanding of how to actually solve the problem. For instance, it's reasonable to expect the boundary between the POS tagger, parser and entity recogniser to disappear, in the same way that spaCy doesn't feature a separate chunker or sentence boundary detector. I read these annotations off the parse tree.
- bobbyi_settv 11y ago
  A library that is constantly changing and removing "dead" code is a library nobody can rely on for production applications. At first the complaint about NLTK was that it was too academic and not appropriate for real-world code, but no real-world code is going to rely on an unreliable library that keeps changing how it works.
  - syllogism 11y ago
    You can maintain the API while overhauling the models underneath. spaCy so far has had almost no API breakages. For instance, you get sentences as follows: doc = nlp(u'Hello world. This is a document.') for sent in doc.sents: for word in sent: ... It doesn't matter to users whether behind the scenes, the sentence boundaries are being calculated from character heuristics, or from the syntactic parse. It was the former, now it's the latter. Similarly, part-of-speech tags are currently predicted in their own processing step. In future they may be predicted jointly with the parsing. The API won't change. Other libraries ask users to choose between a variety of different statistical models, e.g. they ask you to specify that you want the "neural network dependency parser", or the "probabilistic context-free grammar parser", or whatever. By doing this they tie the API to those models. spaCy just picks the best one and gives it to you. The benefit is that you don't need to be informed when a new model is implemented, even if the change is quite drastic. The modelling is a transient implementation detail, not exposed in the API.
- bobbyi_settv 11y ago
deleted 11y ago
[deleted]

by Radim 11y ago

nkozyra 11y ago

NLTK is expansive, which seems to be the real, primary complaint. But it's also not $5,000 a year. I applaud the effort, and the product looks very good, but I don't think there was enough here to say, definitively, "don't use NLTK."

TD-Linux 11y ago
If you missed it, SpaCy is AGPLv3 as well. $5,000 a year doesn't seem so bad if you are doing something that can't work with the AGPLv3.
- nkozyra 11y ago
  No, I saw that ... my point is that there's a profit motive that would make utilization of NLTK a non starter regardless of either platform's merits.
- eridius 11y ago
  There's a pretty large gap between "$5,000 a year is reasonable" and "AGPLv3 is usable" (or any GPL variant). Speaking personally, the overwhelming majority of projects I work on can't ever go near GPL, because the project itself does not want to catch the awful GPL virus. Even open-source projects, nearly all of the ones I touch are not GPL but are some other, much more sane, license like Apache, BSD, or MIT.
  - belorn 11y ago
    No one here is interested in reading inflammatory posts about tiresome old flame wars. You call it an awful virus, I call proprietary licenses a distasteful cancer. You call it insane, I call it entitled attitude to other peoples work. At work and private life I don't include other peoples copyrighted work if I don't have a license for it. It's one simple rule. If I don't want to agree to the license because its too expensive or it put requirements on me which I refuse (like NDA's), then I have a simple choose to make. I can chose not use it, I can implement it myself, or I can hire someone else to do it.
    - eridius 11y ago
      You're arguing the wrong point. I'm not advocating for ignoring licenses, and I have no idea why you think I would. My point is entirely that the OP's claim that $5,000 a year if you can't use the AGPL is absolutely fucking stupid. There's tons of reasons why you can't use the AGPL and can't afford $5,000 a year. So go take your sanctimony elsewhere.
      - belorn 11y ago
        If you don't want to pay the price and refuse to accept the conditions in which it is given to you for free, then tough luck!. Don't come to me crying about how you want it for free while adding your own license on top because your work is special.
        eridius 11y ago
        You really suck at reading comprehension. Please stop with this nonsense, it has no bearing whatsoever to any of my comments.
        belorn 11y ago
        You seem very angry at people who offer you free code to use. Please don't post inflammatory comments in HN in that state.

aloedrip 11y ago

Could not have said this better. New developers should not discount the benefits of hindsight, and humility goes a long way in the open source community.

danieldk 11y ago

It's easier to tout "simplicity and leanness" when you don't have to carry over all the baggage and backward compatibility accumulated over the years. Well, why do we have to build large, clunky (NLP) libraries to start with? Build lean and mean components as UNIX programs or easily bindable libraries and use a reasonable input/output format. E.g. nearly every statistical dependency parser uses CoNLL-X for input/output. You'll have no trouble swapping out MaltParser, Turbo Parser, or my neural net dependency parser. They all use the same, boring, tabular format. Sure, this could be more work for a beginner. So, a project could make a curated list / meta-package of components that are robust, state-of-the-art and work together.

syllogism 11y ago
I definitely agree that the NLP library shouldn't have an opinion about your machine learning, shouldn't give you every accuracy measure under the sun, etc. My original vision for spaCy was smaller. But there are two problems. 1) If I ship you a statistical model, and it's late in a pipeline, like a parser, the earlier components in the pipeline are not swappable. If you change the tokenization, POS tagging, lemmatization etc, the parser model will give you worse output. This isn't obvious to people, and the problem can be subtle. For instance, some NER models use POS tag features, others don't. 2) The output format isn't actually that convenient. It sucks that everyone has to write this tree processing code, and then aligning the tokenized output back to the original string is a pain, if you want to calculate mark-up.
Radim 11y ago
We're in violent agreement. It makes me sad to see neat, focused libraries have their mission blurred, API expanded, code base obfuscated... Until they satisfy everybody's use case, which is to say, they're useless. Some features are best left to user-land. It's a non-trivial tradeoff, obviously: "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can." (Zawinski's law) I actually wrote a rant on this topic a while ago: http://rare-technologies.com/data-streaming-in-python-generators-iterators-iterables/ http://rare-technologies.com/data-streaming-in-python-genera...

Smerity 11y ago

As you state from looking at the website, the stated purpose is for education, but NLTK is used (and advertised) as far more than that. Due to this mixed message, NLTK is used in places it shouldn't be - the author's main concern. For me, NLTK also has issues in education. Teaching requires clarity - a complex codebase rarely allows for that. The author wrote an article "Parsing English in 500 lines of Python"[1] which does a great job of explaining, by being simple and lean, how to parse. Additionally, it achieved the same level of accuracy as the Stanford NLP parser - a larger and more complex parser. That to me is the pinnacle of an educational objective - clear, concise, and practical. [1]: http://spacy.io/blog/parsing-english-in-python/ http://spacy.io/blog/parsing-english-in-python/

syllogism 11y ago

> Will SpaCy's fate be different, 10 years down the road? Yes --- because I consolidate my algorithms and delete dead code. I've probably written five or six times as much code as currently lives in spaCy. I hope by then spaCy will be smaller, not bigger, as we reach a more concise understanding of how to actually solve the problem. For instance, it's reasonable to expect the boundary between the POS tagger, parser and entity recogniser to disappear, in the same way that spaCy doesn't feature a separate chunker or sentence boundary detector. I read these annotations off the parse tree.

bobbyi_settv 11y ago
A library that is constantly changing and removing "dead" code is a library nobody can rely on for production applications. At first the complaint about NLTK was that it was too academic and not appropriate for real-world code, but no real-world code is going to rely on an unreliable library that keeps changing how it works.
- syllogism 11y ago
  You can maintain the API while overhauling the models underneath. spaCy so far has had almost no API breakages. For instance, you get sentences as follows: doc = nlp(u'Hello world. This is a document.') for sent in doc.sents: for word in sent: ... It doesn't matter to users whether behind the scenes, the sentence boundaries are being calculated from character heuristics, or from the syntactic parse. It was the former, now it's the latter. Similarly, part-of-speech tags are currently predicted in their own processing step. In future they may be predicted jointly with the parsing. The API won't change. Other libraries ask users to choose between a variety of different statistical models, e.g. they ask you to specify that you want the "neural network dependency parser", or the "probabilistic context-free grammar parser", or whatever. By doing this they tie the API to those models. spaCy just picks the best one and gives it to you. The benefit is that you don't need to be informed when a new model is implemented, even if the change is quite drastic. The modelling is a transient implementation detail, not exposed in the API.
bobbyi_settv 11y ago

deleted 11y ago

[deleted]