4 ms·

Disappointed to see zero mention of related work, and no comparison against the many other papers that do transition-based SRL, many of which can produce arbitr

Disappointed to see zero mention of related work, and no comparison against the many other papers that do transition-based SRL, many of which can produce arbitrary output graphs. The authors use a transformation of the PropBank data, so their numbers aren't directly comparable against the literature. That's okay --- the PropBank format itself isn't that great, and we'd like a better balance of usability and learnability. However there have been dozens of systems published against a different graph transformation of the PropBank data, e.g. the CoNLL 2008 and 2009 shared task: http://ufal.mff.cuni.cz/conll2009-st/#task http://ufal.mff.cuni.cz/conll2009-st/#task There should be no problem to compare these systems against the system in this paper. The unfair comparison would be to take a system that produces PropBank output, and transform its output into this format. This comparison would be unfair because the system would be optimising a different measure. A fairer comparison would be to take a system that can produce arbitrary graphs, and train it on the authors' format. The transition system looks quite interesting, and pretty easy to try -- so I'm actually tempted to implement this and give it a go. However with no meaningful evaluation we have no way of knowing whether it outperforms the many alternatives that have been proposed, e.g. https://www.semanticscholar.org/search?year%5B%5D=2017&year%5B%5D=2017&q=incremental%20semantic%20parser&sort=relevance https://www.semanticscholar.org/search?year%5B%5D=2017&year%... The accompanying paper is described as a tech report, so arguably it shouldn't be held to the same standard. But still. I'd have hoped to see more leadership from the anchor author Fernando Perreira -- who has written plenty of relevant literature on this, starting in the 80s. The field is already very difficult to follow, even if papers are more forthcoming about how they fit into previous work. Without that, the claims in each paper become very difficult to understand. I would therefore suggest rejection of this paper if I received it for blind review.

kleiba 9y ago
It's especially disappointing as the author of SEMAFOR, Dipanjan Das, now works for Google too.
nl 9y ago
I think this comes across as a little too critical. I agree with everything you have said. But to me it looks like this came from an industrial implementation which they decided to open source. That means they weren't aiming for SOTA on standard datasets, but for what worked best on their internal benchmarks. The evaluation procedures for the published report looks like it was derived from how they train it internally. That means they probably had to justify all the time they were spending on open sourcing it, and writing a tech report. So while it would be great to see standard datasets tried, I much prefer seeing the code open sourced. Also, nothing stopping someone else publishing a nice short paper on the CoNLL datasets using this software.
- syllogism 9y ago
  I'm not sure why you're thinking this was an industrial implementation? It's built on DRAGNN, which is pretty new, so that doesn't give much time for it to be in production. Also, if it were in production the blog post would surely say that -- instead it's quite clearly marked as an experimental system, for research. The lead author seems to be a researcher, too.
  - nl 9y ago
    The whole thing feels like it is bits of code extracted or rewritten out of an existing system. Take the whole Myelin JIT compiler. That's a pretty weird thing to build just for research into frame parsing.

kleiba 9y ago

It's especially disappointing as the author of SEMAFOR, Dipanjan Das, now works for Google too.

nl 9y ago

I think this comes across as a little too critical. I agree with everything you have said. But to me it looks like this came from an industrial implementation which they decided to open source. That means they weren't aiming for SOTA on standard datasets, but for what worked best on their internal benchmarks. The evaluation procedures for the published report looks like it was derived from how they train it internally. That means they probably had to justify all the time they were spending on open sourcing it, and writing a tech report. So while it would be great to see standard datasets tried, I much prefer seeing the code open sourced. Also, nothing stopping someone else publishing a nice short paper on the CoNLL datasets using this software.

syllogism 9y ago
I'm not sure why you're thinking this was an industrial implementation? It's built on DRAGNN, which is pretty new, so that doesn't give much time for it to be in production. Also, if it were in production the blog post would surely say that -- instead it's quite clearly marked as an experimental system, for research. The lead author seems to be a researcher, too.
- nl 9y ago
  The whole thing feels like it is bits of code extracted or rewritten out of an existing system. Take the whole Myelin JIT compiler. That's a pretty weird thing to build just for research into frame parsing.