3 ms·

Yep, currently the breaking stories are identified in the 15 minute intervals, so all the duplicate stories showing are from different times. If they have the s

Yep, currently the breaking stories are identified in the 15 minute intervals, so all the duplicate stories showing are from different times. If they have the same title that means that I can train the model to combine them. They also could be press releases, considering the same title was found for multiple articles over the past 30-45 minutes. The data pipeline currently removes duplicate titles, but only inside single batches. So, it could be a long sustained press release.. like propaganda. As of writing it shows three different stories though: - 12:15 Will there be a coronavirus relief bill with $1,200 checks? - 12:30 Nigeria protesters break curfew amid gunfire, chaos in Lagos - 12:30 Google antitrust case to turn on how search engine grew dominant Maybe what you're seeing is a bug in the website client, I'll investigate it.

awestroke 6y ago
If they have the exact same title, perhaps you can skip having a deep neural net and just do a string comparison? Might be sligthtly faster ;) Finding duplicate stories that have different texts and titles sounds like something you should use AI for
- prohobo 6y ago
  I'm definitely thinking of ways to improve it for long-term stories. You're right, fuzzy string matching would do wonders for that, thanks :)
  - chris_f 6y ago
    I've used trigram matching of article title strings with some success for comparing similarity of news article. If the strings are greater than .6 they are probably about the same topic.
petre 6y ago
There are news about Pakistan and Hungary in Poland.
- prohobo 6y ago
  Most news articles mention multiple locations. It's probable that those pieces talked about both Pakistan/Hungary and Poland, so will be found in both countries. The locations are ordered by salience (proximity to start of the article, number of times mentioned), but not completely filtered, so on this specific visualization you will see a lot of matched locations which you wouldn't consider part of the article, but were mentioned in passing. Trying to understand the content of an article is still really, really difficult, and error prone.

awestroke 6y ago

If they have the exact same title, perhaps you can skip having a deep neural net and just do a string comparison? Might be sligthtly faster ;) Finding duplicate stories that have different texts and titles sounds like something you should use AI for

prohobo 6y ago
I'm definitely thinking of ways to improve it for long-term stories. You're right, fuzzy string matching would do wonders for that, thanks :)
- chris_f 6y ago
  I've used trigram matching of article title strings with some success for comparing similarity of news article. If the strings are greater than .6 they are probably about the same topic.

petre 6y ago

There are news about Pakistan and Hungary in Poland.

prohobo 6y ago
Most news articles mention multiple locations. It's probable that those pieces talked about both Pakistan/Hungary and Poland, so will be found in both countries. The locations are ordered by salience (proximity to start of the article, number of times mentioned), but not completely filtered, so on this specific visualization you will see a lot of matched locations which you wouldn't consider part of the article, but were mentioned in passing. Trying to understand the content of an article is still really, really difficult, and error prone.