3 ms·

In a couple places it sounds like you're interested in scraping HN -- if that's the case there is an official API: https://github.com/HackerNews/API https://git

by paulgb 9y ago

In a couple places it sounds like you're interested in scraping HN -- if that's the case there is an official API: https://github.com/HackerNews/API https://github.com/HackerNews/API My own take on it in general is that for personal/research use I'm not morally opposed to scraping, even when it's in violation of the ToS, with two conditions: that it doesn't place an unreasonable burden on the server, and that it doesn't invade people's privacy. The legal significance of the ToS is murky at best (disclaimer: I'm not a lawyer) but if the site asks you specifically to stop scraping them or puts up a technical barrier you should stop (morally and, in the US at least, legally: see craigslist v 3taps)

jackschultz 9y ago
Ahh thanks, that'll be good to use and seems like HN wouldn't be overall annoyed by me including these comments on the site. And in general as well for other people to run projects with the data.
bigiain 9y ago
I'm pretty much on this wavelength. If I can automate something that I'd otherwise do manually - or that a less technical person would happily do manually, I'm fine with having python or perl and a cron job do it. What I try to avoid is scope creep when my automation makes it easier to magnify the scale. My dad used to collect his local synoptic chart from the weather bureau website - and it frustrated him that they only archived 1 week's worth so when he was away he'd need to ask someone else to download them for him or end up with gaps in his collection - I happily scripted that up for him. What I wouldn't do for him though, was make it grab _every_ chart for every region available - if he wasn't doing that by hand, I wasn't going to break the terms of service that much further and automate it for him... I'm happy to admit when I do things like they they're without doubt technically infringing on someone's legal rights, but ethically I consider this to be in the "of course I'll walks across an intersection against a red light in the middle of the night with no cars around" class of wrongdoing.
anothercomment 9y ago
The full HN dataset is also on Google Big Query: https://cloud.google.com/bigquery/public-data/hacker-news https://cloud.google.com/bigquery/public-data/hacker-news (updated daily)

by paulgb 9y ago

jackschultz 9y ago

Ahh thanks, that'll be good to use and seems like HN wouldn't be overall annoyed by me including these comments on the site. And in general as well for other people to run projects with the data.

bigiain 9y ago

I'm pretty much on this wavelength. If I can automate something that I'd otherwise do manually - or that a less technical person would happily do manually, I'm fine with having python or perl and a cron job do it. What I try to avoid is scope creep when my automation makes it easier to magnify the scale. My dad used to collect his local synoptic chart from the weather bureau website - and it frustrated him that they only archived 1 week's worth so when he was away he'd need to ask someone else to download them for him or end up with gaps in his collection - I happily scripted that up for him. What I wouldn't do for him though, was make it grab _every_ chart for every region available - if he wasn't doing that by hand, I wasn't going to break the terms of service that much further and automate it for him... I'm happy to admit when I do things like they they're without doubt technically infringing on someone's legal rights, but ethically I consider this to be in the "of course I'll walks across an intersection against a red light in the middle of the night with no cars around" class of wrongdoing.

anothercomment 9y ago

The full HN dataset is also on Google Big Query: https://cloud.google.com/bigquery/public-data/hacker-news https://cloud.google.com/bigquery/public-data/hacker-news (updated daily)