2 ms·
HTML scraping should be the last option you consider to get data after all else fails. Even though the Hacker News API (https://github.com/HackerNews/API https
by minimaxir 7y ago
HTML scraping should be the last option you consider to get data after all else fails.
Even though the Hacker News API (https://github.com/HackerNews/API https://github.com/HackerNews/API) is somewhat old, it's a much more kosher way of getting data.
Even better is to use the public data dump in BigQuery (https://console.cloud.google.com/marketplace/details/y-combinator/hacker-news https://console.cloud.google.com/marketplace/details/y-combi...). Quick query to get all top-level comments in posts by whoishiring:
#standardSQL
WITH whoishiring_posts AS (
SELECT id from `bigquery-public-data.hacker_news.full`
WHERE `by`="whoishiring" AND type="story"
)
SELECT text
FROM `bigquery-public-data.hacker_news.full`
WHERE type="comment"
AND parent IN (SELECT id from whoishiring_posts)
- plibither8 7y agoThe API that https://hn.algolia.com https://hn.algolia.com provides can also be used! In my experience, it's more verbose and the queries are more customisable.
- saagarjha 7y agoThis is great! I've been wondering for a while how to efficiently load a comment tree without spamming the Hacker News API with hundreds of requests, and this seems like it would work nicely.
- ChristianBundy 7y agoExample Node.js library that I wrote a few years ago: https://github.com/fraction/node-hacker-news-api https://github.com/fraction/node-hacker-news-api
- minimaxir 7y agoAgreed, unfortunately.
- Naac 7y agoBut the official Hacker News API is really painful to use. Specifically because each comment is its own API call (?! ). I know the HN staff mentioned they're working on a V2 API, but until then the HN HTML tree is so clean, I think this is an exceptional case where scraping the HTML yields a cleaner result that's also less load on the HN servers. EDIT: Like the other comment on this thread mentioned, using https://hn.algolia.com/api https://hn.algolia.com/api seems ideal here.
- deleted 7y ago[deleted]
- santiagobasulto 7y agoI second this. And it's true that the official HN API is painful to use. Algolia's is a lot friendlier. I wrote a thin Python library on top: https://github.com/santiagobasulto/python-hacker-news https://github.com/santiagobasulto/python-hacker-news Here's an example: results = search_by_date( 'Who is hiring?', ask_hn=True, author='whoishiring', hits_per_page=1000) Full working example: https://gist.github.com/santiagobasulto/b3a3bb94a26c1798b9650232ed1ade07 https://gist.github.com/santiagobasulto/b3a3bb94a26c1798b965...
- inc_eax 7y agoI am aware of the ease of use aspect when it comes to using BigQuery, however, I do believe that it is paid to use service.
- minimaxir 7y agoYou get 1TB/month free. The query above only takes 6GB.