3 ms·

They're all online written news sources, but I believe GDELT is also getting into TV news so that might be coming in the future. There is no official list curr

They're all online written news sources, but I believe GDELT is also getting into TV news so that might be coming in the future. There is no official list currently, but you could query either GDELT or TidalWaves to see which are available in the data sets. I could add a page to the marketing website which displays all of them, instead of the top sources for the latest news - would that be something useful for people?

wombatmobile 6y ago
Thanks for replying. I'm not sure what is useful yet - first of all I am trying to understand the scope of the data sources and the methodology for selections displayed in TidalWaves. I would like to see the list of publications if that is possible. I am wondering what "trending stories" means in TidalWaves? For example, if the NYT at any point contains 500 stories, and is refreshed say 3 times every 24 hours, with say 100 stories aged off and 100 new stories added each addition, what is a trending story in the NYT? Does the front page headline count more than a page 27 minor traffic report? How is the top 5 trending stories for USA derived?
- prohobo 6y ago
  I just did a quick query grouping all sources and number of articles per source in the database (currently about 4 days worth of data): https://pastebin.com/HyktFpML https://pastebin.com/HyktFpML As you can see, there are already almost 8000 sources. I'm honestly not sure how GDELT scrapes its data though, but this is what they do for over 5 years now and it's backed by Google Cloud processing, so I'm sure it's ever expanding and very thorough. The kinds of articles which are discovered revolve around world events, however. So you won't see random blog posts, or traffic reports. Instead you'll see geopolitics, big tech news, and cultural discourse. Trending stories are found for the entire set of articles in a batch, with no separation for source, category, or location. If the top stories all happen to be in the USA in a given moment, then that's all you'll find. If NYT happens to publish an article that fits an overall story being talked about in the rest of the batch, it'll get added to that story. Trending popularity is decided by how many individual sources talk about the given topic. This is good enough to identify the Zeitgeist of a moment, but it likely won't catch nuances in evolving stories.
  - wombatmobile 6y ago
    Thanks for compiling that list. I have had a quick look and downloaded it for a longer look. Some GDELT methodology is extolled here: https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-realtime/ https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-...

wombatmobile 6y ago

Thanks for replying. I'm not sure what is useful yet - first of all I am trying to understand the scope of the data sources and the methodology for selections displayed in TidalWaves. I would like to see the list of publications if that is possible. I am wondering what "trending stories" means in TidalWaves? For example, if the NYT at any point contains 500 stories, and is refreshed say 3 times every 24 hours, with say 100 stories aged off and 100 new stories added each addition, what is a trending story in the NYT? Does the front page headline count more than a page 27 minor traffic report? How is the top 5 trending stories for USA derived?

prohobo 6y ago
I just did a quick query grouping all sources and number of articles per source in the database (currently about 4 days worth of data): https://pastebin.com/HyktFpML https://pastebin.com/HyktFpML As you can see, there are already almost 8000 sources. I'm honestly not sure how GDELT scrapes its data though, but this is what they do for over 5 years now and it's backed by Google Cloud processing, so I'm sure it's ever expanding and very thorough. The kinds of articles which are discovered revolve around world events, however. So you won't see random blog posts, or traffic reports. Instead you'll see geopolitics, big tech news, and cultural discourse. Trending stories are found for the entire set of articles in a batch, with no separation for source, category, or location. If the top stories all happen to be in the USA in a given moment, then that's all you'll find. If NYT happens to publish an article that fits an overall story being talked about in the rest of the batch, it'll get added to that story. Trending popularity is decided by how many individual sources talk about the given topic. This is good enough to identify the Zeitgeist of a moment, but it likely won't catch nuances in evolving stories.
- wombatmobile 6y ago
  Thanks for compiling that list. I have had a quick look and downloaded it for a longer look. Some GDELT methodology is extolled here: https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-realtime/ https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-...