5 ms·
Ask HN: HN data access
How can I get access to all of the submissions on news.ycombinator.com? I don't want the comments, just the posts.
It seems like after every HNS outage settings get added to make it more difficult to access the content on HNS. For example, it seems like you have to sign in now to access older posts. Also, the thrift database has only about 4million records but the hids on HNS are in the 5millions.
- unimpressive 14y agoI think that there are some mirrors floating around. But I wouldn't know where those mirrors are, or what kind of access they allow.
- bashgrep 14y agoWhere did you read about them?
- unimpressive 14y agoOn the original millionshort thread, where somebody pointed out the number of HN mirrors and linked a few.
- bashgrep 14y agoThese all lead to dead ends: https://news.ycombinator.com/item?id=1721105 https://news.ycombinator.com/item?id=1721105 https://news.ycombinator.com/item?id=1881262 https://news.ycombinator.com/item?id=1881262 Do you have a link for the post you are talking about? EDIT: In this thread pg says wait 30 seconds between each request (https://news.ycombinator.com/item?id=1702488 https://news.ycombinator.com/item?id=1702488), but that doesn't work either. EDIT: "unimpressive" Was this is what you were referencing?: https://docs.google.com/spreadsheet/ccc?key=0AqL8kR005z0QdENvNUJJTjYxY2lVa0RqUzJhTHFqT0E&authkey=CIeUndcL&hl=en&authkey=CIeUndcL#gid=0 https://docs.google.com/spreadsheet/ccc?key=0AqL8kR005z0QdEN...
- unimpressive 14y agohttps://news.ycombinator.com/item?id=3911687 https://news.ycombinator.com/item?id=3911687 It appears to be dead. EDIT: Another one that appears to be among the living. http://hackerbra.in/ http://hackerbra.in/
- Mz 14y agoI am not sure, but I think the IDs count posts and comments. So I think that means about 4 million submissions and over a million comments. Not all submissions draw comments. Can anyone verify this?
- pasbesoin 14y agoLook at the URLs for a submission (found via its "discuss" link on the post listing page) and for a non-post comment. They are identical in format; only the ID value varies. For this to work, they must share the same set of ID values.
- bashgrep 14y agoSeems like HNS is rate limiting connection speed for connections from amazon ec2 machines. HNS/pg, do you not want us scraping HNS? What is the best way to get all the posts?