4 ms·

‘Reverse Engineering’ Hacker News into an API

52 points by rashadphil 4y ago

bdcravens 4y ago
(2018)
- timoteostewart 4y ago
  Bottom of article says August 2nd, 2017 even
dr_pardee 4y ago
Search hackd in iOS app store
- deleted 4y ago
  [deleted]
- CharlesW 4y ago
  No results. First two results are "Jack'd" and "Grindr", followed by many more dating apps.
deleted 4y ago
[deleted]
CharlesW 4y ago
Related: https://swizec.com/blog/how-i-reverseengineered-hacker-news/ https://swizec.com/blog/how-i-reverseengineered-hacker-news/
exikyut 4y ago
The main issue with all apps like this is not the reverse-engineering, which is relatively straightforward, but the fact that HN sits behind Cloudflare, with the WAF and ratelimiting enabled. It's not possible to achieve capability-parity just *reading* HN as the API does not return comment karma, which is used to sort the view. (I think this is the subtler half of why HN reader sites never catch on - the interesting comments are never at the top, so the presented content is very noticeably that much more boring.) So a while back I tried building a simple scraper component (an internal API, I guess) that would use Firebase for realtime updates, and strategically poll the site (say one request every 3-5 seconds) to get the current sort order. I never got to the Firebase part because I crashed into Cloudflare while fiddling around tuning the ratelimiting. So yeah. You need full Chrome (to satisfy whatever Cloudflare's WAF throws at you) and a captcha-solving subscription (because the WAF will get the flyswat out when it sees your datacenter IP) to be able to do this properly - especially if you want to APIify this sort of thing.
- timoteostewart 4y ago
  I recently wrote a HN reader and decided to forgo handling comments for reasons you mention, including the obscurity of comment karma scores. I focused on the story browsing side of things, and the links to comment sections take you straight to HN’s comment page. Link to my app if anyone’s curious. Red question mark in upper left corner of page goes to its “about” page: https://www.thnr.net https://www.thnr.net
- livinglist 4y ago
  Actually if I recall correctly, even though official api doesn’t show Karma value of comment, the children comment IDs (kids is the field name) of each comment are actually sorted by karma (or whatever method HN uses to sort on server side)
splittingTimes 4y ago
For years, i just want to be able to easily search through my "favorite submissions" which I use as a kind of bookmark. Guess we will never see that. Not on the website nor in an app.
- yesenadam 4y ago
  Did you try writing to the HN mod(s) asking about that? (At the address given on the guidelines page) Sounds like a good idea.
- rashadphil 4y ago
  I actually setup my personal website to track my favorite submissions. https://rashad.vercel.app/hackernews https://rashad.vercel.app/hackernews. Favorite submissions can be made public and are easily scrape-able. What do you mean by easily search through?
captn3m0 4y ago
Didn’t HN get an official API a long time ago? Or was that read-only?
- guessmyname 4y ago
  Link at the bottom of HackerNews that says “API” points to this GitHub repository [1]. The first commit is from Oct 2, 2014 while the article was published on April 1, 2018. However, it seems that you did not read the first paragraph of the article which states the following: > While building hackd I faced a problem - the official Hacker News API doesn’t allow for interaction, such as upvoting, posting and commenting. I wanted hackd to be a full featured Hacker News client, so this wasn’t going to cut it. So to answer your question: > Didn’t HN get an official API a long time ago? Or was that read-only? Yes, it appears that at the time of publication, the API was read-only. [1] https://github.com/HackerNews/API https://github.com/HackerNews/API
  - capableweb 4y ago
    What do you mean "at the time of publication"? It's still read-only, unless I'm missing something obvious.
stefanvdw1 4y ago
Replicating the features that require being logged in such as upvoting and commenting is actually not trivial. I recently built a Hacker News client [0] on top of the official API and using the method described in the article to login users is very inconsistent. I kept running into Captcha issues and even when that didn’t happen, sometimes a login request would simply not return the expected token. Unless the official API starts offering user login, this will remain difficult. 0. https://modernorange.io https://modernorange.io
searchableguy 4y ago
I tried creating a GraphQL API[0] for HN for both read and write operations so I could eventually use it to write a HN client app but I hit a wall due to HN scraping policy. It limits you to fetching a page once every 30 seconds[1]. Yes, I confirmed with dang that I would need to adhere to this. [0] https://github.com/searchableguy/hackernews_graphql https://github.com/searchableguy/hackernews_graphql [0] https://hngraphql.fly.dev/graphql https://hngraphql.fly.dev/graphql [1] https://news.ycombinator.com/ https://news.ycombinator.com/

52 points by rashadphil 4y ago

bdcravens 4y ago

(2018)

timoteostewart 4y ago
Bottom of article says August 2nd, 2017 even

dr_pardee 4y ago

Search hackd in iOS app store

deleted 4y ago
[deleted]
CharlesW 4y ago
No results. First two results are "Jack'd" and "Grindr", followed by many more dating apps.

deleted 4y ago

[deleted]

CharlesW 4y ago

Related: https://swizec.com/blog/how-i-reverseengineered-hacker-news/ https://swizec.com/blog/how-i-reverseengineered-hacker-news/

exikyut 4y ago

The main issue with all apps like this is not the reverse-engineering, which is relatively straightforward, but the fact that HN sits behind Cloudflare, with the WAF and ratelimiting enabled. It's not possible to achieve capability-parity just *reading* HN as the API does not return comment karma, which is used to sort the view. (I think this is the subtler half of why HN reader sites never catch on - the interesting comments are never at the top, so the presented content is very noticeably that much more boring.) So a while back I tried building a simple scraper component (an internal API, I guess) that would use Firebase for realtime updates, and strategically poll the site (say one request every 3-5 seconds) to get the current sort order. I never got to the Firebase part because I crashed into Cloudflare while fiddling around tuning the ratelimiting. So yeah. You need full Chrome (to satisfy whatever Cloudflare's WAF throws at you) and a captcha-solving subscription (because the WAF will get the flyswat out when it sees your datacenter IP) to be able to do this properly - especially if you want to APIify this sort of thing.

timoteostewart 4y ago
I recently wrote a HN reader and decided to forgo handling comments for reasons you mention, including the obscurity of comment karma scores. I focused on the story browsing side of things, and the links to comment sections take you straight to HN’s comment page. Link to my app if anyone’s curious. Red question mark in upper left corner of page goes to its “about” page: https://www.thnr.net https://www.thnr.net
livinglist 4y ago
Actually if I recall correctly, even though official api doesn’t show Karma value of comment, the children comment IDs (kids is the field name) of each comment are actually sorted by karma (or whatever method HN uses to sort on server side)

splittingTimes 4y ago

For years, i just want to be able to easily search through my "favorite submissions" which I use as a kind of bookmark. Guess we will never see that. Not on the website nor in an app.

yesenadam 4y ago
Did you try writing to the HN mod(s) asking about that? (At the address given on the guidelines page) Sounds like a good idea.
rashadphil 4y ago
I actually setup my personal website to track my favorite submissions. https://rashad.vercel.app/hackernews https://rashad.vercel.app/hackernews. Favorite submissions can be made public and are easily scrape-able. What do you mean by easily search through?

captn3m0 4y ago

Didn’t HN get an official API a long time ago? Or was that read-only?

guessmyname 4y ago
Link at the bottom of HackerNews that says “API” points to this GitHub repository [1]. The first commit is from Oct 2, 2014 while the article was published on April 1, 2018. However, it seems that you did not read the first paragraph of the article which states the following: > While building hackd I faced a problem - the official Hacker News API doesn’t allow for interaction, such as upvoting, posting and commenting. I wanted hackd to be a full featured Hacker News client, so this wasn’t going to cut it. So to answer your question: > Didn’t HN get an official API a long time ago? Or was that read-only? Yes, it appears that at the time of publication, the API was read-only. [1] https://github.com/HackerNews/API https://github.com/HackerNews/API
- capableweb 4y ago
  What do you mean "at the time of publication"? It's still read-only, unless I'm missing something obvious.

stefanvdw1 4y ago

Replicating the features that require being logged in such as upvoting and commenting is actually not trivial. I recently built a Hacker News client [0] on top of the official API and using the method described in the article to login users is very inconsistent. I kept running into Captcha issues and even when that didn’t happen, sometimes a login request would simply not return the expected token. Unless the official API starts offering user login, this will remain difficult. 0. https://modernorange.io https://modernorange.io

searchableguy 4y ago

I tried creating a GraphQL API[0] for HN for both read and write operations so I could eventually use it to write a HN client app but I hit a wall due to HN scraping policy. It limits you to fetching a page once every 30 seconds[1]. Yes, I confirmed with dang that I would need to adhere to this. [0] https://github.com/searchableguy/hackernews_graphql https://github.com/searchableguy/hackernews_graphql [0] https://hngraphql.fly.dev/graphql https://hngraphql.fly.dev/graphql [1] https://news.ycombinator.com/ https://news.ycombinator.com/