5 ms·
This is the side-project, that I have been building recently. The API allows searching over the data from many news sites and news aggregators, it also allows t
by caballeto 6y ago
This is the side-project, that I have been building recently. The API allows searching over the data from many news sites and news aggregators, it also allows to easily extract whole news sites. I built it for two reasons. First, I wanted to have a source of recent news data to play with, essentially to run data analysis and train ml models on. Secondly, because alternative services had wildly high pricing, like 400-500$ per month, and I thought I could build and deploy such a system for a much cheaper price and improve my programming skills along the way. I would very much appreciate any comments/suggestions about the serive or how to make it better :)
- sbr464 6y agoNice work. Will share feedback soon after testing.
- caballeto 6y agoGreat, thanks, would appreciate it very much. I am in process of thinking about new features, so if you have some special use cases, I could think about that.
- sbr464 6y agoSeemed pretty straightforward after a few tests. The only thing I noticed was regarding unique identifiers, the URL in the sources endpoint didn't seem to exactly match the URL attached to a headline? I could be wrong though, and not a big issue. Was more thinking about data cleanup and linking data easily (article/sources/authors etc).
- caballeto 6y agoThanks very much for the feedback. Yes, I think your point is right. For some domains, I try to match the url with sources from "sources" list, and for others I extract source field directly from the url, I think there is a small bug when doing the former, thanks for pointing out. I will fix it soon.
- bvm 6y agoHow are you licensing the content?
- jwhiz22 6y agoHe's not.
- caballeto 6y agoRight, I will definitely work on adding data policy to terms of service. I think it will go along the lines, that you cannot remove the attributions/trademarks, if you are republishing the data, but doing analytics on it is ok.
- bvm 6y agoIt's a pretty fine line you're treading here. A lot of the reason the competition is so expensive is that they are licensing the data in bulk. There's lots of litigation in this area, see the Meltwater cases in US and UK.
- caballeto 6y agoIf you have say 10k news sources from tens of countries around the globe, I doubt that it would be feasible to contract with every specific site out there, especially if you get data say from Google News.
- overcast 6y agoThe prices are high because you’re paying for licensing costs.
- todsacerdoti 6y agoI would love to integrate your product into Pipedream.com so developers can quickly integrate Datanews with other apps such as Slack, Discord, Reddit, Google Sheets, etc. Please reach out -- tod at pipedream.
- caballeto 6y agoGreat idea, I like it.