5 ms·

First of all: great work! Even though I think OpenNLP should also be mentioned, since it's released under the commercially-friendly Apache 2 license, it's great

First of all: great work! Even though I think OpenNLP should also be mentioned, since it's released under the commercially-friendly Apache 2 license, it's great that you provide this. Also, I think it is a shame that science lost you (I assume) ;). I think the other interesting problem to tackle currently is training data. The situation for English is ok, if you have a couple of thousand of dollars to spare for a commercial license (which may be problematic for bootstrapping). But for many other languages there aren't even treebanks available that can be used for commercial purposes. It would be great if some annotation project started that aimed to provide annotations under a liberal license. (Ps. I have a statistical dependency parser written in Go, which I will probably release soon in case anyone is interested ;).)

syllogism 12y ago
Well, I'm distributing trained models with this. Users shouldn't need to retrain unless they're doing research, in which case they should have access to the data. I agree that the data situation is troubling, though. I don't understand why Google gave the English Web Treebank to the LDC. Why not just distribute it themselves? The LDC is really more of a problem than a help now. For instance, the OntoNotes corpus costs $0 for non-commercial use. Great! How do you get it? Send the LDC a fax, and when they get around to it, they send you a log in to their ancient website. It used to be a valuable service to host and distribute this data. Now, this is no longer really the case, but it's still standard to distribute via them.
- danieldk 12y ago
  Well, I'm distributing trained models with this. Users shouldn't need to retrain unless they're doing research, in which case they should have access to the data. I haven't read LDC's license on Penn Treebank recently, but AFAIR you cannot just redistribute models that were trained on the Penn Treebank. Or put differently, you can distribute the model, but any users still have to obtain a license for the treebank. That's why we are still stuck with the Brown corpus and such. I don't understand why Google gave the English Web Treebank to the LDC. Why not just distribute it themselves? Indeed.
  - danieldk 12y ago
    I haven't read LDC's license on Penn Treebank recently, Just checked the license and my memory seems to be correct: https://catalog.ldc.upenn.edu/license/ldc-non-members-agreement.pdf https://catalog.ldc.upenn.edu/license/ldc-non-members-agreem... :(
    - syllogism 12y ago
      This is the non-commercial agreement --- I'm licensing the resources commercially.
- covi 12y ago
  Great work! I'm curious (for academic research), what's the "3 billion word corpus" that you trained the estimates with? Is it available somewhere?
ris 12y ago
"since it's released under the commercially-friendly Apache 2 license" I do hate the proliferation of the idea of BSD-style licenses being "commercially friendly". You can do one hell of a lot of things commercially with GPL software. It's only under very specific circumstances that you are obliged to release source for GPL software. And as (I would wager probably) ~99% of NLP software is used internal to a company and never distributed as such, the differentiation between licensing models would never actually come into play.
- gojomo 12y ago
  But this is AGPL, which massively broadens the circumstances where you must release source.

syllogism 12y ago

Well, I'm distributing trained models with this. Users shouldn't need to retrain unless they're doing research, in which case they should have access to the data. I agree that the data situation is troubling, though. I don't understand why Google gave the English Web Treebank to the LDC. Why not just distribute it themselves? The LDC is really more of a problem than a help now. For instance, the OntoNotes corpus costs $0 for non-commercial use. Great! How do you get it? Send the LDC a fax, and when they get around to it, they send you a log in to their ancient website. It used to be a valuable service to host and distribute this data. Now, this is no longer really the case, but it's still standard to distribute via them.

danieldk 12y ago
Well, I'm distributing trained models with this. Users shouldn't need to retrain unless they're doing research, in which case they should have access to the data. I haven't read LDC's license on Penn Treebank recently, but AFAIR you cannot just redistribute models that were trained on the Penn Treebank. Or put differently, you can distribute the model, but any users still have to obtain a license for the treebank. That's why we are still stuck with the Brown corpus and such. I don't understand why Google gave the English Web Treebank to the LDC. Why not just distribute it themselves? Indeed.
- danieldk 12y ago
  I haven't read LDC's license on Penn Treebank recently, Just checked the license and my memory seems to be correct: https://catalog.ldc.upenn.edu/license/ldc-non-members-agreement.pdf https://catalog.ldc.upenn.edu/license/ldc-non-members-agreem... :(
  - syllogism 12y ago
    This is the non-commercial agreement --- I'm licensing the resources commercially.
covi 12y ago
Great work! I'm curious (for academic research), what's the "3 billion word corpus" that you trained the estimates with? Is it available somewhere?

ris 12y ago

"since it's released under the commercially-friendly Apache 2 license" I do hate the proliferation of the idea of BSD-style licenses being "commercially friendly". You can do one hell of a lot of things commercially with GPL software. It's only under very specific circumstances that you are obliged to release source for GPL software. And as (I would wager probably) ~99% of NLP software is used internal to a company and never distributed as such, the differentiation between licensing models would never actually come into play.

gojomo 12y ago
But this is AGPL, which massively broadens the circumstances where you must release source.