4 ms·

While on the "how HN works" topic, I have a question: I'm working on an HN app reader idea that I think is unique (more to come) that requires regularly gettin

by dchuk 9y ago

While on the "how HN works" topic, I have a question: I'm working on an HN app reader idea that I think is unique (more to come) that requires regularly getting the front page posts and comments. I wrote a script to do that through the Firebase API but holy shit it ended up needing thousands of requests to get all the data for the current state of the front page. So instead, I wrote a scraper script to produce the same thing, just with 30 requests (1 per front page post). (Trying to turn these damn <tr>'s into a recursive comment tree was a quite the mindfuck btw) ...is it ok to scrape HN? I see nothing in the robots.txt to say it shouldn't be allowed, and I actually feel better making 30 scraping requests rather than XXXX API requests. EDIT: Also happy to receive suggestions. I'm coding in Ruby and don't see an obvious way to access the Firebase DB directly (quite frankly I don't even know if what I just said makes sense) so any help is appreciated.

deleted 9y ago
[deleted]
tbirdz 9y ago
The algolia api has more bulk operations, so you can get more than one response per api request. Also if you look around, there are some datasets where people have downloaded all the HN API json responses, and have made them available, so you could use those for local development and testing. As far as ethics go, it's probably better to use the API than scraping. Sure you're making more requests, but I think the firebase servers and cdn have far more ability to handle a lot of requests than the HN server does. If having that many requests would put firebase out in any significant way, then I'm sure they would put a query load limit on the HN api.
- Artemis2 9y ago
  HN is behind Cloudflare; if you're not logged in the pages can be cached so I wouldn't expect small-scale scrapping to have any effect. Cloudflare might stop you though.
cperciva 9y ago
For Hacker News Daily I make requests to https://hacker-news.firebaseio.com/v0/ https://hacker-news.firebaseio.com/v0/ IIRC I made this change at the suggestion of someone at YC (maybe dang, but it might have been before he took over HN) so I think it's very likely that this is the route they would prefer you to use. But if in doubt, send them an email; dang is very responsive.
- i336_ 9y ago
  Hmmm. What would you recommend for the instance where someone's emailed hn@ycombinator.com and hasn't gotten a response? I suspect it's something I said, I'm just unsure exactly what. I asked about the posting and API ratelimits, the reviewer idea mentioned at https://news.ycombinator.com/item?id=11662380 https://news.ycombinator.com/item?id=11662380, source release (notwithstanding the voting system, which is fine), and other curios. I don't think I said anything offensive. I have successfully gotten in touch in the past, so I'm just not sure what to do. The email was sent on the 27th of April; I just checked my spam folder (I must admit I didn't until now...), but I don't think gmail would have deleted any incorrectly-filed replies, since they'd be less than a month old. I'm not complaining or trying to create drama, just curious what to do next. My fear is that it's not possible to answer the "what did I do" question due to social context and so forth (a hole I've fallen into many, many times due to communication perception issues).
  - i336_ 9y ago
    Update for the record: I was, very happily, completely wrong. dang got in touch to let me know the delay was due to backlog and that he'll properly reply when he gets the chance. Absolutely fine by me; I'm super glad to know my email was fine :D
chrisjh 9y ago
Happy to help - email is in my profile.
- swampthinker 9y ago
  Am I crazy? I don't see an email in your profile.
  - webmaven 9y ago
    It is in the 'about' field.
    - grzm 9y ago
      It wasn't at the time your parent posted. It is there now. That, or we're experiencing folie à deux.
ibejoeb 9y ago
Check out my implementation of comments parsing from the html, which is used in HNES: https://github.com/ibejoeb/HNES/commit/054fc4e626137aadef6f0857099efa34cbd7835e https://github.com/ibejoeb/HNES/commit/054fc4e626137aadef6f0...
- dchuk 9y ago
  Nice! I basically did it the same way. I'll push my code up to a public repo soon and let you know to compare.
cyberferret 9y ago
The Firebase API is the way to go. I built a small app to do exactly as you described [0] and wrote a blog post about it [1]. Drop me a line if you need any more info. UPDATE: Using the Firebase feed means you don't have to do periodic scraping. You can simply set a listener for the FB change() event and the API will basically tell you when there is fresh information from the front page etc. [0] - https://tophn.info https://tophn.info [1] - https://hackernoon.com/tophn-a-fun-side-project-built-with-vue-js-and-rethinkdb-f22159641c1b https://hackernoon.com/tophn-a-fun-side-project-built-with-v...
- dchuk 9y ago
  I can't seem to find anything that would let me do what you did, but with Ruby. I'm not opposed to using Node, just prefer Ruby more.
iKlsR 9y ago
hckrnews.com ?
neoncontrails 9y ago
It sounds like your struggles are with Firebase more than the actual scraping part, so I'm not sure how useful this will be. But there's a marvelous demonstration of Clojure's ability to render and serve a scraped digest of the HackerNews homepage. https://github.com/swannodette/enlive-tutorial/blob/master/readme.textile https://github.com/swannodette/enlive-tutorial/blob/master/r... The final product is about 17 lines of code. Which is quite compact, sure—but if it looks like Perl, who cares? Thankfully, it looks nothing like Perl. The code is deeply intuitive and evangelizes the idea that sending an object over a network can often be defined as a composition of functions resulting in the creation of the very object that we're sending. Whether that object is an HTML document, JSON, or whatever, the alluring premise is that these can all be forged from a single bulk-concatenation of strings returned by functions being unrolled in an inner-loop. It looks practically effortless. Might give you some ideas for batching those 30 GET requests. (Or maybe not—I've never used firebase before!)
- dchuk 9y ago
  Very cool! Always wanted to get into Clojure, I just don't code for a living anymore so only do it in my free time so I tend to turn to tools I know. Bookmarked though!
jjt-yn_t 9y ago
While on the "how HN works" topic, where is the button/page to submit a "Ask YN" ??? Not a neophyte by any means, but what is to me a thorough web investigation came up empty. Just me?

by dchuk 9y ago

deleted 9y ago

[deleted]

tbirdz 9y ago

The algolia api has more bulk operations, so you can get more than one response per api request. Also if you look around, there are some datasets where people have downloaded all the HN API json responses, and have made them available, so you could use those for local development and testing. As far as ethics go, it's probably better to use the API than scraping. Sure you're making more requests, but I think the firebase servers and cdn have far more ability to handle a lot of requests than the HN server does. If having that many requests would put firebase out in any significant way, then I'm sure they would put a query load limit on the HN api.

Artemis2 9y ago
HN is behind Cloudflare; if you're not logged in the pages can be cached so I wouldn't expect small-scale scrapping to have any effect. Cloudflare might stop you though.

cperciva 9y ago

For Hacker News Daily I make requests to https://hacker-news.firebaseio.com/v0/ https://hacker-news.firebaseio.com/v0/ IIRC I made this change at the suggestion of someone at YC (maybe dang, but it might have been before he took over HN) so I think it's very likely that this is the route they would prefer you to use. But if in doubt, send them an email; dang is very responsive.

i336_ 9y ago
Hmmm. What would you recommend for the instance where someone's emailed hn@ycombinator.com and hasn't gotten a response? I suspect it's something I said, I'm just unsure exactly what. I asked about the posting and API ratelimits, the reviewer idea mentioned at https://news.ycombinator.com/item?id=11662380 https://news.ycombinator.com/item?id=11662380, source release (notwithstanding the voting system, which is fine), and other curios. I don't think I said anything offensive. I have successfully gotten in touch in the past, so I'm just not sure what to do. The email was sent on the 27th of April; I just checked my spam folder (I must admit I didn't until now...), but I don't think gmail would have deleted any incorrectly-filed replies, since they'd be less than a month old. I'm not complaining or trying to create drama, just curious what to do next. My fear is that it's not possible to answer the "what did I do" question due to social context and so forth (a hole I've fallen into many, many times due to communication perception issues).
- i336_ 9y ago
  Update for the record: I was, very happily, completely wrong. dang got in touch to let me know the delay was due to backlog and that he'll properly reply when he gets the chance. Absolutely fine by me; I'm super glad to know my email was fine :D

chrisjh 9y ago

Happy to help - email is in my profile.

swampthinker 9y ago
Am I crazy? I don't see an email in your profile.
- webmaven 9y ago
  It is in the 'about' field.
  - grzm 9y ago
    It wasn't at the time your parent posted. It is there now. That, or we're experiencing folie à deux.

ibejoeb 9y ago

Check out my implementation of comments parsing from the html, which is used in HNES: https://github.com/ibejoeb/HNES/commit/054fc4e626137aadef6f0857099efa34cbd7835e https://github.com/ibejoeb/HNES/commit/054fc4e626137aadef6f0...

dchuk 9y ago
Nice! I basically did it the same way. I'll push my code up to a public repo soon and let you know to compare.

cyberferret 9y ago

The Firebase API is the way to go. I built a small app to do exactly as you described [0] and wrote a blog post about it [1]. Drop me a line if you need any more info. UPDATE: Using the Firebase feed means you don't have to do periodic scraping. You can simply set a listener for the FB change() event and the API will basically tell you when there is fresh information from the front page etc. [0] - https://tophn.info https://tophn.info [1] - https://hackernoon.com/tophn-a-fun-side-project-built-with-vue-js-and-rethinkdb-f22159641c1b https://hackernoon.com/tophn-a-fun-side-project-built-with-v...

dchuk 9y ago
I can't seem to find anything that would let me do what you did, but with Ruby. I'm not opposed to using Node, just prefer Ruby more.

iKlsR 9y ago

hckrnews.com ?

neoncontrails 9y ago

It sounds like your struggles are with Firebase more than the actual scraping part, so I'm not sure how useful this will be. But there's a marvelous demonstration of Clojure's ability to render and serve a scraped digest of the HackerNews homepage. https://github.com/swannodette/enlive-tutorial/blob/master/readme.textile https://github.com/swannodette/enlive-tutorial/blob/master/r... The final product is about 17 lines of code. Which is quite compact, sure—but if it looks like Perl, who cares? Thankfully, it looks nothing like Perl. The code is deeply intuitive and evangelizes the idea that sending an object over a network can often be defined as a composition of functions resulting in the creation of the very object that we're sending. Whether that object is an HTML document, JSON, or whatever, the alluring premise is that these can all be forged from a single bulk-concatenation of strings returned by functions being unrolled in an inner-loop. It looks practically effortless. Might give you some ideas for batching those 30 GET requests. (Or maybe not—I've never used firebase before!)

dchuk 9y ago
Very cool! Always wanted to get into Clojure, I just don't code for a living anymore so only do it in my free time so I tend to turn to tools I know. Bookmarked though!

jjt-yn_t 9y ago

While on the "how HN works" topic, where is the button/page to submit a "Ask YN" ??? Not a neophyte by any means, but what is to me a thorough web investigation came up empty. Just me?