3 ms·

Agreed, it seems they really hand-picked some shining examples for this post, and it would have been more interesting to see the full spectrum of when it works

by pierrec 10y ago

Agreed, it seems they really hand-picked some shining examples for this post, and it would have been more interesting to see the full spectrum of when it works and when it doesn't. Perhaps the README in the Github repo is a bit more honest in terms of representativeness, though it only has 4 examples, one of them is an interesting failure: article: novell inc. chief executive officer eric schmidt has been named chairman of the internet search-engine company google . human: novell ceo named google chairman machine: novell chief executive named to head internet company https://github.com/tensorflow/models/tree/master/textsum https://github.com/tensorflow/models/tree/master/textsum (It's still great that it beats all competition on the ROUGE score, of course.)

hisham_hm 10y ago
I don't see that as a failure. It did produce a sentence that is shorter, grammatical (though "named to head" is a bit weird) and essentially true — calling Google an "internet company" would make sense in its early days (back when Google would be prefixed by "internet search-engine company").
- spicyj 10y ago
  Most humans would interpret "novell chief executive named to head internet company" to mean "novell is an internet company and its chief executive just became its head" which is incorrect (and a little nonsensical since the CEO already is in change).
  - nl 10y ago
    That's pretty interesting. It's taken me 5 readings of that sentence, including once out-loud to get your reading of it. I thought the generated summary was really, really good. But I knew that Novell wasn't considered an internet company, so it wasn't until I made myself ignore that before I could see the other reading.
    - spicyj 10y ago
      Oh, that's funny. I was thinking it was. I guess it's a software company but not an internet company.
- Cybiote 10y ago
  I didn't think it was a failure either until I realized that I was letting future knowledge leak into the past! There is more than one internet company so upon reading that headline, given that it must be a novel event, my question would be: "Which Company?". Now I have to read the article until I find out. The human summary is better because I don't have to ask that question.
Cybiote 10y ago
I think the holiday pay example is more glaring. Seen in isolation, I would be confused as to what on earth it was getting at. Furthermore, the summary is no good. The abstract isn't either. My summary would be: British Gas continues to fight eu court's decision that commission be included in holiday pay. Continue is used here to emphasize that the case is not yet over. On the other hand, the football summary is exemplary; better than the provided abstract.

by pierrec 10y ago

hisham_hm 10y ago

I don't see that as a failure. It did produce a sentence that is shorter, grammatical (though "named to head" is a bit weird) and essentially true — calling Google an "internet company" would make sense in its early days (back when Google would be prefixed by "internet search-engine company").

spicyj 10y ago
Most humans would interpret "novell chief executive named to head internet company" to mean "novell is an internet company and its chief executive just became its head" which is incorrect (and a little nonsensical since the CEO already is in change).
- nl 10y ago
  That's pretty interesting. It's taken me 5 readings of that sentence, including once out-loud to get your reading of it. I thought the generated summary was really, really good. But I knew that Novell wasn't considered an internet company, so it wasn't until I made myself ignore that before I could see the other reading.
  - spicyj 10y ago
    Oh, that's funny. I was thinking it was. I guess it's a software company but not an internet company.
Cybiote 10y ago
I didn't think it was a failure either until I realized that I was letting future knowledge leak into the past! There is more than one internet company so upon reading that headline, given that it must be a novel event, my question would be: "Which Company?". Now I have to read the article until I find out. The human summary is better because I don't have to ask that question.

Cybiote 10y ago

I think the holiday pay example is more glaring. Seen in isolation, I would be confused as to what on earth it was getting at. Furthermore, the summary is no good. The abstract isn't either. My summary would be: British Gas continues to fight eu court's decision that commission be included in holiday pay. Continue is used here to emphasize that the case is not yet over. On the other hand, the football summary is exemplary; better than the provided abstract.