3 ms·
I think this comes across as a little too critical. I agree with everything you have said. But to me it looks like this came from an industrial implementation
by nl 9y ago
I think this comes across as a little too critical.
I agree with everything you have said. But to me it looks like this came from an industrial implementation which they decided to open source.
That means they weren't aiming for SOTA on standard datasets, but for what worked best on their internal benchmarks. The evaluation procedures for the published report looks like it was derived from how they train it internally.
That means they probably had to justify all the time they were spending on open sourcing it, and writing a tech report.
So while it would be great to see standard datasets tried, I much prefer seeing the code open sourced.
Also, nothing stopping someone else publishing a nice short paper on the CoNLL datasets using this software.
- syllogism 9y agoI'm not sure why you're thinking this was an industrial implementation? It's built on DRAGNN, which is pretty new, so that doesn't give much time for it to be in production. Also, if it were in production the blog post would surely say that -- instead it's quite clearly marked as an experimental system, for research. The lead author seems to be a researcher, too.
- nl 9y agoThe whole thing feels like it is bits of code extracted or rewritten out of an existing system. Take the whole Myelin JIT compiler. That's a pretty weird thing to build just for research into frame parsing.