4 ms·
Thanks! There are a lot of issues with GDELT data, but the things that come to mind recently are: - Missing keywords. Huge topics like "brexit" won't be found,
by prohobo 6y ago
Thanks! There are a lot of issues with GDELT data, but the things that come to mind recently are:
- Missing keywords. Huge topics like "brexit" won't be found, so you need to extract those yourself.
- Dirty titles. You need to extract the article titles from the website metadata, and more often than not they'll pre/append their site name like "[title] - DailyStormer" and links like "[title]: Business, Stocks, News" etc. That requires NLP to fix.
- Non-standardized locations. Location tags are all over the place, so "united_states" might have many tags like "us", "usa", "america", "the_states", etc. I'm still working on combining these tags in the API.
- Grimm1 6y agoOkay so the data is messy and they have a limited set of acknowledged keywords and entity types so will have to work those out ourselves. Got it. That is super helpful information thank you!
- panarky 6y agoUgh, "DailyStormer" is your example of a "news" site? https://www.splcenter.org/fighting-hate/extremist-files/individual/andrew-anglin https://www.splcenter.org/fighting-hate/extremist-files/indi...
- prohobo 6y agoI was just being cheeky. Their articles don't get pulled into the data, but I'm sure there are plenty of other bad sources if you look around.