caballeto

8 ms·

31.
by caballeto 6y ago
Hi all, this is the side project that I am currently working on. Datanews Monitoring API gathers hundreds of thousands news articles per day and allows you to easily monitor them for keywords you are interested in. Essentially, I see severa
32.
by caballeto 6y ago
This is the third result when searching for "Dropbox joke". But what is a Dropbox joke?
33.
by caballeto 6y ago
Starred yours :DD
34.
Gnews – minimalistic JavaScript library for Google News scraping(github.com)
15 points by caballeto 6y ago|3 comments
35.
by caballeto 6y ago
I've recently needed a JS library for scraping Google News, but the ones I've found used outdated Google News APIs. So, I have created this minimalistic, but rather powerful package, for asynchronous extraction of Google News data
36.
by caballeto 6y ago
There is a project along these lines at Google, which is used to test graphic drivers. The generated code is randomly (but equivalently) transformated, and the result is verified to be the same. If it doesn't match, then there is a bug
37.
by caballeto 6y ago
The tech stack is Python, Java, Spring, Elasticsearch, Redis/MySQL. This is a beta version with explicit support for 50 sources + Google News aggregator. I plan to release new version in about a week, which will support another 400-500
38.
by caballeto 6y ago
Great idea, I like it.
39.
by caballeto 6y ago
I use standard web requests to get raw html and parse the data from it, no headless browsing. I guess all the news pages I've met so far were static.
40.
by caballeto 6y ago
Hi, thanks for feedback. I am going to push changes to display only a snippets from the articles soon.
41.
by caballeto 6y ago
Hi, just finished implementing this stuff. To search a AND b use 'q=A+B'. Here are a few examples: > curl -XGET -G 'api.datanews.io/v1/news' -d 'apiKey=API_KEY' --data-urlencode 'q="Europ
42.
by caballeto 6y ago
Thanks, will take take a note for future.
43.
by caballeto 6y ago
Hi, thanks for the feedback! So far, I haven't implemented this feature yet. The query parameter gets tokenized as is, and then tokens are used to search in the index. Could you provide more details about your use case? As I understand
44.
by caballeto 6y ago
If you have say 10k news sources from tens of countries around the globe, I doubt that it would be feasible to contract with every specific site out there, especially if you get data say from Google News.
45.
by caballeto 6y ago
That's interesting. I've browsed a few of alternative apis, and if I get it correctly then newsapi.org returns full text articles for paid plans. I would research this question further, and maybe will change the api to return only
46.
by caballeto 6y ago
Thanks very much for the feedback. Yes, I think your point is right. For some domains, I try to match the url with sources from "sources" list, and for others I extract source field directly from the url, I think there is a small
47.
by caballeto 6y ago
Thanks for the feedback! Concerning rate limiting, I think I may have placed rules a bit too strict, as the system is hosted on AWS Free Tier. There is definitely a room for increase in RPS per user, but I wanted to play safe.
48.
by caballeto 6y ago
Right, I will definitely work on adding data policy to terms of service. I think it will go along the lines, that you cannot remove the attributions/trademarks, if you are republishing the data, but doing analytics on it is ok.
49.
by caballeto 6y ago
First, I extract the links to the news articles by searching them on the home page or sitemaps. Then, for each article I scrap raw html and extract key things like title, description, publish date, authors, etc. Usually each news article ha
50.
by caballeto 6y ago
This is a great question. Thus far I thought, that the content that is publicly available without any limitation (e.g. membership access), can be scraped by anyone. You can take a look at hiQ Labs vs. LinkedIn [1]. LinkedIn's public da
51.
by caballeto 6y ago
No, I don't use RSS feeds. I extract news links from the home page and sitemaps mainly. Most of the news articles have a news schema embedded within them. You can check info about it at schema.org. This is a piece of JSON which contain
52.
by caballeto 6y ago
The scraper only scraps the recent content, so it avoids scraping same links. As the daily number of articles published by the website is not very big, I make requests with a break in between, total number of requests is not that big, so th
53.
by caballeto 6y ago
Great, thanks, would appreciate it very much. I am in process of thinking about new features, so if you have some special use cases, I could think about that.
54.
Show HN: News Searching API(datanews.io)
40 points by caballeto 6y ago|42 comments
55.
by caballeto 6y ago
This is the side-project, that I have been building recently. The API allows searching over the data from many news sites and news aggregators, it also allows to easily extract whole news sites. I built it for two reasons. First, I wanted t
56.
by caballeto 6y ago
Dart VM is a good example. Their JIT is frequently faster than the AOT compiled code, although they may improve the AOT speed in future releases.
57.
by caballeto 6y ago
Great read, thanks! I would definitely consider using dedicated servers instead of VPCs. AWS seems to be ridiculously overpriced.
58.
by caballeto 6y ago
Could you please elaborate why? I thought they should be premium, as they are faster, have more memory and better performance because of bare metal, etc.
59.
by caballeto 6y ago
That seems like a very powerful machine at an incredibly low cost. The closest equivalent at DigitalOcean is RAM 64 GB, 16 CPUs, 1.25 TB SSD, 9 TB transfer at $320/month. If I understand correctly the DO instance will have lower perfor
60.
Ask HN: How do you learn big codebase fast?
1 points by caballeto 6y ago|0 comments

More ›