3 ms·

It's easier to tout "simplicity and leanness" when you don't have to carry over all the baggage and backward compatibility accumulated over the years. Well, wh

It's easier to tout "simplicity and leanness" when you don't have to carry over all the baggage and backward compatibility accumulated over the years. Well, why do we have to build large, clunky (NLP) libraries to start with? Build lean and mean components as UNIX programs or easily bindable libraries and use a reasonable input/output format. E.g. nearly every statistical dependency parser uses CoNLL-X for input/output. You'll have no trouble swapping out MaltParser, Turbo Parser, or my neural net dependency parser. They all use the same, boring, tabular format. Sure, this could be more work for a beginner. So, a project could make a curated list / meta-package of components that are robust, state-of-the-art and work together.

syllogism 11y ago
I definitely agree that the NLP library shouldn't have an opinion about your machine learning, shouldn't give you every accuracy measure under the sun, etc. My original vision for spaCy was smaller. But there are two problems. 1) If I ship you a statistical model, and it's late in a pipeline, like a parser, the earlier components in the pipeline are not swappable. If you change the tokenization, POS tagging, lemmatization etc, the parser model will give you worse output. This isn't obvious to people, and the problem can be subtle. For instance, some NER models use POS tag features, others don't. 2) The output format isn't actually that convenient. It sucks that everyone has to write this tree processing code, and then aligning the tokenized output back to the original string is a pain, if you want to calculate mark-up.
Radim 11y ago
We're in violent agreement. It makes me sad to see neat, focused libraries have their mission blurred, API expanded, code base obfuscated... Until they satisfy everybody's use case, which is to say, they're useless. Some features are best left to user-land. It's a non-trivial tradeoff, obviously: "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can." (Zawinski's law) I actually wrote a rant on this topic a while ago: http://rare-technologies.com/data-streaming-in-python-generators-iterators-iterables/ http://rare-technologies.com/data-streaming-in-python-genera...

syllogism 11y ago

I definitely agree that the NLP library shouldn't have an opinion about your machine learning, shouldn't give you every accuracy measure under the sun, etc. My original vision for spaCy was smaller. But there are two problems. 1) If I ship you a statistical model, and it's late in a pipeline, like a parser, the earlier components in the pipeline are not swappable. If you change the tokenization, POS tagging, lemmatization etc, the parser model will give you worse output. This isn't obvious to people, and the problem can be subtle. For instance, some NER models use POS tag features, others don't. 2) The output format isn't actually that convenient. It sucks that everyone has to write this tree processing code, and then aligning the tokenized output back to the original string is a pain, if you want to calculate mark-up.

Radim 11y ago

We're in violent agreement. It makes me sad to see neat, focused libraries have their mission blurred, API expanded, code base obfuscated... Until they satisfy everybody's use case, which is to say, they're useless. Some features are best left to user-land. It's a non-trivial tradeoff, obviously: "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can." (Zawinski's law) I actually wrote a rant on this topic a while ago: http://rare-technologies.com/data-streaming-in-python-generators-iterators-iterables/ http://rare-technologies.com/data-streaming-in-python-genera...